Adaptive  Inventory  Control  for  Non- Stationary  Demand 


WITH  Partial  Information 


James  Thomas  Treharne 


Certificate  of  Approval: 


Harriet  Black  Nembhard 

Assistant  Professor 

Industrial  and  Systems  Engineering 


Tohn  F.  Pritchett 
Dean 

Graduate  School ' 


Adaptive  Inventory  Control  for  Non-Stationary  Demand 


WITH  Partial  Information 


James  Thomas  Treharne 


A  Dissertation 
Submitted  to 
the  Graduate  Faculty  of 
Auburn  University 
in  Partial  Fulfillment  of  the 
Requirements  for  the 
Degree  of 

Doctor  of  Philosophy 


DISTRIBUTION  STATEMENT  A 

Approved  for  Public  Release 
Distribution  Unlimited 


Auburn,  Alabama 
June  11,  1999 


19990520  003 


Vita 


James  Thomas  Treharne,  son  of  Richard  T.  Treharne  and  Elizabeth  J.  Treharne, 
was  born  July  28,  1957  in  Lakewood,  Ohio.  He  attended  Adlai  E.  Stevenson  High  School 
in  Livonia,  Michigan.  He  then  graduated  from  the  United  States  Military  Academy  in 
1979  and  was  commissioned  as  a  Regular  Army  2nd  Lieutenant  in  the  United  States 
Army  Corps  of  Engineers.  He  was  an  Assistant  Professor  of  Military  Science  at  Auburn 
University  from  1986-1990.  In  1991  he  earned  a  Master  of  Science  in  Industrial  Engi¬ 
neering  at  Auburn  University  and  was  inducted  into  Phi  Kappa  Phi.  His  thesis  was 
entitled,  “RSSP-  A  Fortran  Simulation  Package  For  Use  In  Teaching  Response  Surface 
Methodology.”  He  then  served  as  a  military  operations  research  analyst  with  the  Army 
Concepts  Analysis  Agency  between  1991  and  1993.  He  is  a  1994  graduate  of  the  United 
States  Army  Command  and  General  Staff  College.  LTC  Treharne  is  married  to  the  for¬ 
mer  Patricia  A.  Broaddus  of  Winchester,  Kentucky.  They  have  six  children,  Rebecca 
Joy,  James  David,  Elizabeth  Mae,  Rachel  Lynn,  Julia  Ann,  and  Lydia  Grace. 


Dissertation  Abstract 


Adaptive  Inventory  Control  for  Non-Stationary  Demand 
WITH  Partial  Information 

James  Thomas  Treharne 

Doctor  of  Philosophy,  June  11,  1999 
(M.S.,  Auburn  University,  1991) 

(B.S.,  United  States  Military  Academy,  1979) 

145  Typed  Pages 
Directed  by  Charles  R.  Sox 

This  dissertation  presents  optimal  and  suboptimal  procedmres  to  solve  inventory 
control  problems  that  have  non-stationary  demand  and  partial  information.  In  each 
period,  the  underlying  demand  distribution  may  change  according  to  a  known  Markov 
process.  The  problem  is  characterized  as  partial  information  because  some  parameter 
of  the  demand  probability  distribution  is  not  known  with  certainty;  however,  there  is 
a  known  prior  distribution  for  the  unknown  parameter.  In  one  case,  there  is  a  proba¬ 
bility  density  function  for  the  demand  that  has  at  least  one  unknown  parameter,  but 
this  parameter  has  a  known  probability  distribution.  In  another  case,  there  is  a  set 
of  candidate  demand  probability  distributions.  The  parameter  which  indicates  which 
demand  is  in  effect  at  any  given  time  is  unknown,  but  has  a  known  probability  mass 
function.  The  control  strategies  are  adaptive  because  the  controllers  learn  information 
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about  these  unknown  parameters  over  time  and  adapt  accordingly.  Because  of  the  com¬ 
plexity  of  these  problems,  managers  often  estimate  the  unknown  parameters  and  make 
decisions  assuming  the  estimate  is  correct.  The  computational  results  presented  in  this 
dissertation  demonstrate  that  there  exist  efficient  and  effective  optimal  and  suboptimal 
procedures  to  solve  these  problems  that  potentially  provide  large  cost  savings  compared 
with  this  current  practice.  The  control  strategies  include  open-loop  feedback  and  limited 
look-ahead  control  for  a  finite  horizon  problem,  which  are  compared  to  optimal  and  cer¬ 
tainty  equivalence  control  policies.  A  grid  approximation  and  upper  and  lower  bounds 
for  an  infinite  horizon  problem  are  also  developed. 
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Chapter  1 


Introduction 

The  purpose  of  this  dissertation  is  to  investigate  a  fundamental  inventory  control 
problem  in  which  the  demand  distribution  is  non-stationary  and  the  information  about 
the  distribution  is  not  complete.  We  title  this  dissertation  “Adaptive  Inventory  Con¬ 
trol  for  Non-Stationary  Demand  with  Partial  Information”  because  we  are  interested 
in  examining  such  problems  in  which  we  receive  state-dependent  information  about  the 
underlying,  but  unknown  demand  distribution.  The  system  is  adaptive  because  each 
period  information  is  learned  about  the  value  of  the  unknown  parameter.  The  controller 
uses  this  information  as  it  accumulates  to  adjust  futvire  decisions  accordingly.  Therefore, 
stocking  decisions  are  made  under  the  assumption  that  new  information  dependent  on 
the  current  state  will  be  observed  in  each  time  period,  and  that  this  new  information 
will  be  incorporated  into  future  decisions.  Such  an  adaptive  control  policy  is  important 
because  the  demand  distribution  may  change  each  period.  The  current  business  envi¬ 
ronment  is  marked  by  two  important  characteristics.  The  first  is  that  demand  in  today’s 
markets  is  often  highly  volatile.  This  volatility  is  due  to  a  number  of  factors,  includ¬ 
ing  the  proliferation  of  products,  the  global  marketplace,  shortened  product  life-cycles, 
and  the  rapid  growth  of  technology.  Manufacturers  may  no  longer  safely  assume  that 
product  demand  will  remain  stationary  for  long  periods  of  time.  The  second  charac¬ 
teristic  is  that  we  are  now  inundated  with  vast  amounts  of  information.  Despite  this 
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wealth  of  information,  managers  still  face  a  great  deal  of  uncertainty  about  demand. 
At  the  same  time,  we  now  have  the  computer  capability  to  solve  problems  with  greater 
degrees  of  complexity.  Thus,  we  must  be  able  to  use  all  available  information  to  make 
the  best  inventory  decision,  knowing  that  in  the  next  period  circumstances  will  be  dif¬ 
ferent  and  new  information  will  be  available.  Managers  often  disregard  uncertainty  by 
using  point  forecasts  as  if  they  are  accurate.  Because  of  the  advances  of  computational 
speed  and  the  results  reported  in  this  dissertation,  such  simplifying  assumptions  are  no 
longer  necessary  for  eflScient  and  effective  inventory  control  decisions.  Our  goal  is  to 
show  that  there  are  effective  and  efficient  procedures  to  solve  problems  when  the  de¬ 
mand  distribution  is  non-stationary  and  only  partial  information  is  available.  Generally, 
inventory  problems  can  be  divided  into  four  classes.  The  first  class  is  stationary  demand 
with  complete  information.  The  second  class  is  non-stationary  demand  with  complete 
information.  The  third  class  is  stationary  demand  with  partial  information.  The  final 
class  is  non-stationary  demand  with  partial  information.  Problems  with  complete  infor¬ 
mation  about  the  current  demand  state  have  been  studied  fairly  comprehensively.  The 
introduction  of  partial  information  significantly  increases  the  difficulty  of  the  problem. 
By  partial  information  we  mean  that  the  demand  distribution  possesses  an  unknown 
parameter  that  may  be  either  discrete  or  continuous.  Some  information  in  the  form  of 
a  prior  distribution  is  known  about  the  unknown  parameter.  Very  little  work  has  been 
done  when  the  demand  distribution  is  non-stationary  and  also  has  partial  information 


about  the  current  distribution. 
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To  achieve  our  stated  goal  we  present  in  Chapter  2  a  comprehensive  review  of 
adaptive  inventory  control  for  stationary  demand.  We  provide  a  basic  model  that  can 
represent  many  realistic  problems  in  this  problem  class.  Currently,  this  class  of  problems 
is  not  well  addressed  in  the  literature.  We  show  two  approaches  for  this  type  of  prob¬ 
lem.  In  the  first  approach,  the  demand  distribution  is  assumed  to  belong  to  a  known 
family  but  have  at  least  one  unknown  parameter,  which  itself  has  a  known  probability 
distribution  which  is  called  the  prior  distribution.  The  second  approach  is  to  assume 
that  the  there  is  a  discrete  set  of  possible  demand  distributions  and  a  prior  belief  for 
which  distribution  is  in  effect.  In  Chapter  2  we  present  both  a  model  as  well  as  many  of 
its  characteristics.  We  also  present  some  results  for  optimal  strategies  and  their  prop¬ 
erties.  In  many  cases,  suboptimal  solutions  will  be  necessary,  and  we  discuss  several 
such  suboptimal  control  strategies.  We  discuss  such  key  concepts  as  suflacient  statistics, 
conjugate  families,  censored  versus  uncensored  observations,  and  state  spaice  reduction. 

In  Chapter  3  we  extend  our  model  to  the  case  of  non-stationary  demand  with  partial 
information.  Very  little  work  has  been  done  in  this  important  area.  Managers,  espe¬ 
cially  in  these  cases,  will  often  estimate  or  forecast  the  unknown  parameters  and  assume 
they  are  completely  known.  By  doing  so,  they  are  ignoring  a  significant  amount  of  the 
uncertainty  in  the  system,  and  are  potentially  making  costly  decisions.  We  show  in  this 
chapter  that  there  are  many  other  efficient  suboptimal  control  policies  that  provide  excel¬ 
lent  results  by  incorporating  more  of  the  inherent  uncertainty  into  the  decision  process. 
Fully  considering  uncertainty,  even  over  a  limited  horizon,  may  lead  to  solutions  that  are 
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significantly  better  than  certainty  equivalence  control  methods.  In  both  Chapters  3  and 
4  we  model  the  process  as  a  composite-state,  partially  observed  Markov  decision  process. 

In  Chapter  4  we  continue  to  study  the  non-stationary  problem  with  partial  infor¬ 
mation,  and  we  present  valid  techniques  to  compute  bounds  and  approximations  for  this 
problem.  In  this  chapter,  we  assume  that  the  problem  is  one  with  a  discounted,  infinite 
horizon  because  many  practical  problems  can  be  accurately  modeled  as  infinite  horizon 
problems.  Again,  we  model  the  demand  process  as  a  composite-state,  partially  observed 
Markov  decision  process,  and  we  extend  known  techniques  for  the  stationary  problem. 
Most  problems  do  not  have  a  tractable  optimal  solution  because  the  probability  state 
space  is  uncountably  infinite.  However,  we  can  accurately  approximate  the  optimal  cost 
function  for  some  problems  using  a  grid  approximation.  The  approximation  is  asymp¬ 
totically  exact  as  the  grid  spacing  approaches  zero.  However,  a  small  spacing  can  be 
computationally  infeasible.  Because  we  show  that  the  cost  function  is  piece-wise  linear 
and  concave  in  the  probability  state  space,  we  can  compute  subgradients  that  form  an 
upper  bound.  We  can  easily  calculate  these  subgradients  at  each  extreme  point,  and  we 
can  then  examine  the  problem  at  other  values  in  the  probability  state  space  and  itera¬ 
tively  add  subgradients  that  improve  the  upper  bound.  It  is  also  possible  to  calculate  a 
lower  bound  to  this  function.  We  show  both  how  to  calculate  this  lower  boimd  and  how 
to  improve  the  bound  in  a  similar  way.  Finally,  we  demonstrate  some  of  these  techniques 


with  an  example. 


Demand  (x) 


Figure  1.1:  Non-Stationary  Demand  with  Partial  Information 


In  Figure  1.1  we  present  an  example  of  the  situation  that  we  address.  In  this  case 
the  demand  distribution  belongs  to  one  of  three  fully  defined  distributions.  The  demand 
distribution  currently  in  efiect  is  unknown,  although  ttx,,  ttm,  and  tth  represent  the  beliefs 
that  the  true  distribution  is  either  the  low,  medium,  or  high  distribution.  Because  the 
true  distribution  is  unknown,  we  have  partial  information  available.  The  figure  also 
shows  that  in  the  next  period  the  demand  distribution  may  shift  according  to  a  Markov 
process.  In  this  case,  the  probability  that  the  demand  does  not  change  in  the  next 
period  is  0.8  (broad  arrows),  while  the  probability  that  it  moves  to  one  of  the  other  two 
distributions  is  0.1  each  (thin  arrows).  The  objective  of  this  problem  is  to  minimize  the 
total  costs  over  the  given  horizon,  given  the  non-stationary  demand  process  and  partial 
information  about  its  state. 


6 


In  summaxy,  the  following  chapters  demonstrate  viable  solution  procedures  to  solve 
problems  with  partial  information  and  either  stationary  or  non-stationaxy  demand. 
These  techniques  can  potentially  provide  substantial  cost  savings  over  procedures  in 
common  practice  today. 


Chapter  2 


Review  of  Stationary  Demand  and  Partial  Information 

2.1  Introduction 

This  chapter  provides  inventory  managers  valuable  insights  and  theory  for  deter¬ 
mining  control  policies  when  faced  with  partial  information  about  the  demand  process. 
More  specifically,  the  chapter  reviews  key  literature  for  the  problem  of  controlling  the 
inventory  of  a  product  with  partially  observed,  stationary,  random  demand.  Practition¬ 
ers  usually  presume  it  is  necessary  to  model  their  problem  as  a  “complete  information” 
problem  in  order  to  solve  a  realistic  problem.  However,  this  chapter  demonstrates  that 
it  may  be  both  practical  and  worthwhile  to  solve  the  “partial  information”  problem. 
The  demand  process  is  not  completely  observed  but  is  only  partially  observed  through 
the  random  demand  observations.  Because  the  control  decisions  axe  made  with  only 
partial  information  about  the  demand  process,  the  level  of  uncertainty  and  the  cost  of 
suboptimal  decisions  is  much  higher  than  for  most  inventory  problems  considered  in  the 
research  literature.  This  problem  is  an  accurate  representation  of  the  inventory  control 
problems  faced  by  many  organizations.  However,  it  has  not  been  widely  addressed  in 
the  inventory  literature  or  by  existing  decision  support  systems;  therefore,  inventory 
managers  are  forced  to  make  potentially  costly  simplifying  assumptions  when  addressing 
this  challenging  problem.  This  chapter  discusses  the  existing  research  for  this  problem 
and  points  out  its  critical  components  and  areas  for  future  research.  Although  we  do 
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not  provide  an  exhaustive  summary  of  current  research,  we  do  present  some  of  the  most 
critical  existing  research  for  the  problem. 

Many  inventory  control  problems  are  significantly  affected  by  a  lack  of  demand 
information.  We  focus  our  attention  in  this  chapter  to  that  part  of  the  life  cycle  in 
which  demand  is  assumed  stationary.  As  product  life  cycles  continue  to  shrink,  a  larger 
firaction  of  that  fife  cycle  lies  in  the  initial  and  terminal  periods  where  there  is  a  great 
deal  of  uncertainty  in  the  demand  process.  Some  of  this  uncertainty  applies  also  to  the 
stationary  stage  of  the  life  cycle.  Consequently,  the  area  of  our  concern  in  which  demand 
is  considered  stationary  will  be  marked  by  less  precise  information  about  its  unknown 
parameters.  Also,  the  proliferation  of  products  continues  to  dilute  the  historically  stable 
demand  for  individual  products,  leading  to  more  volatile  and  uncertain  demand.  This 
proliferation  also  increases  the  need  for  automated  or  semi-automated  inventory  control 
systems.  The  competition  in  some  markets  for  commodity-like  products  also  creates 
more  volatile,  uncertain  demand  than  is  assumed  in  most  conventional  models.  This 
research  also  has  a  potential  for  broader  impact  on  other  problems  that  involve  partially 
observed,  random  processes.  Such  problems  arise  in  a  variety  of  problem  domains  such  as 
supply  chain  planning  and  control  (Lee  et  al.  [22]),  maintenance  planning,  yield  planning 
and  management,  military  materiel  management,  and  process  control. 

There  are  two  main  approaches  for  modeling  partial  information  inventory  control 
problems.  In  the  first  approach,  the  demand  distribution  is  assumed  to  belong  to  a 
known  distribution  family,  but  at  least  one  parameter  is  unknown.  A  known  a  priori 
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distribution  is  assumed  for  the  unknown  parameter,  and  subsequent  demand  observations 
are  used  to  compute  a  posterior  distribution  for  this  parameter.  In  the  second  approach, 
the  demand  distribution  is  assumed  to  belong  to  a  known  and  discrete  set  of  candidate 
distributions.  A  known,  discrete  a  priori  distribution  is  assumed  for  this  set  which  is 
also  used  along  with  subsequent  observations  to  compute  a  posterior  distribution.  Both 
types  of  models  can  be  solved  optimally  using  stochastic  dynamic  programming  for  only 
small  problems. 

Because  optimal  control  requires  restrictive  assumptions  or  is  computationally  in¬ 
tractable,  most  practical  solutions  rely  on  suboptimal  control.  A  very  common  subopti- 
mal  control  strategy  is  a  type  of  certainty  equivalence  controller.  In  this  type  of  policy, 
the  parameter  estimates  are  generated  from  historical  data  and  then  are  assumed  to  be 
known  with  certainty  in  the  control  phase.  For  example,  they  might  assume  a  Poisson 
distribution  with  a  known  parameter  based  on  last  year’s  demand.  The  parameter  esti¬ 
mates  may  be  revised  periodically,  but  this  approach  takes  no  further  consideration  of 
the  inherent  uncertainty  in  those  estimates.  A  very  important  area  of  investigation  is  to 
determine  when  this  class  of  strategies  is  effective  and  what  other  types  of  strategies  can 
be  efficiently  employed  when  they  are  ineffective. 

The  chapter  is  organized  as  follows.  In  Section  2.2  we  present  a  basic  model  for  this 
problem  and  discuss  some  of  its  characteristics.  The  next  section.  Section  2.3,  presents 
some  results  for  optimal  control  strategies  and  their  properties.  Section  2.4  discusses 
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several  suboptimal  control  strategies,  and  the  last  section  provides  some  concluding 
remarks  and  areas  for  valuable  futmre  research. 

2.2  Basic  Model  and  Insights 

We  first  define  the  basic  inventory  model  and  discuss  some  of  its  important  charac¬ 
teristics.  Any  extensions  or  exceptions  to  this  model  are  indicated  where  they  occur  in 
the  chapter.  The  problem  is  stationary,  that  is,  the  demand  is  independent  and  identi¬ 
cally  distributed  across  periods,  and  the  various  cost  parameters  remain  constant.  Thus, 
our  focus  is  only  on  that  part  of  the  life  cycle  that  is  “stationary” .  However,  the  de¬ 
cision  maker  does  not  have  complete  information  about  the  parameters  of  the  demand 
distribution.  At  least  one  parameter,  r,  is  unknown,  and  this  unknown  parameter  has 
a  known  a  priori  distribution  /(r).  Although  we  use  notation  that  presumes  /  is  con¬ 
tinuous,  the  model  is  easily  extended  to  cases  where  /  is  discontinuous  or  discrete.  The 
planning  horizon  may  be  finite  or  infinite.  Stockouts  are  either  backordered  or  become 
lost  sales.  If  stockouts  are  lost,  the  lost  sales  may  be  observed  or  imobserved.  If  lost 
sales  are  observed,  all  demand  observations  are  exact  and  the  starting  inventory  for  the 
next  period  is  at  least  zero.  The  order  lead  time  is  a  known  constant,  possibly  zero. 
The  procurement  cost  may  include  a  known  fixed  cost.  The  procurement,  shortage  and 
holding  costs  may  be  linear,  convex,  or  non-lineax  functions.  The  current  stock  level  is 
always  known  with  certainty,  and  the  salvage  values  at  the  end  of  the  horizon  may  be 
positive  or  negative.  Stock  cannot  be  freely  disposed  of  although  some  authors  allow 
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this  primarily  to  simplify  the  solution.  Finally,  stock  does  not  deteriorate,  although 
this  can  be  easily  incorporated  into  the  model  when  it  is  a  fixed  proportion  of  on-hand 
inventory.  These  assumptions  make  the  solution  to  the  general  problem  quite  difiicult 
to  solve.  However,  in  many  special  cases  or  under  relaxed  assumptions  there  are  very 
practical  solutions  with  useful  results. 

Figure  (2.1)  describes  the  decision  process.  The  decision  maker  uses  current  infor¬ 
mation  to  choose  an  action,  the  order  quantity.  An  order  that  was  placed  a  lead  time 
in  the  past  is  then  received.  Demand  is  then  observed.  In  the  case  of  unobserved  lost 
sales,  only  actual  sales  axe  observed.  The  period  costs  are  then  assessed.  The  decision 
maker  records  demand  or  sales  data  and  updates  the  available  information  and  repeats 
the  process. 


Figure  2.1:  Decision  Process 


Table  2.1  shows  the  notation  that  is  used  throughout  the  chapter.  Additional  terms 


axe  defined  as  needed. 
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Var 

Description 

Var 

Description 

T 

length  of  planning  horizon 

a 

discount  rate 

Pi-) 

stockout  penalty  cost 

1 

positive  lead  time 

hi-) 

holding  cost 

St 

sufficient  statistic  in  period  t 

c(-) 

procurement  cost 

It 

vector  of  information  in  period  t 

K 

positive  fixed  order  cost 

r 

unknown  parameter 

Xt 

inventory  level  in  period  t 

ftir\It) 

prior  distribution  of  r  in  t 

Ut 

inventory  position 

(piwlSt) 

conditional  demand  in  t 

at 

order  quantity  in  period  t 

vector  of  orders  in  period  t 

Wt 

demand  in  period  t  ^{w]  r) 

4>iw-,  r) 

demand  density 

zt 

sales  in  period  t 

Table  2.1:  Notation 


The  information  available  at  the  beginning  of  period  t,  It,  can  often  be  smnmarized 
by  a  sufficient  statistic,  St.  Such  a  statistic  contains  the  necessary  information  to  compute 
an  optimal  policy  for  period  t.  The  exact  form  of  this  statistic  depends  on  the  specific 
problem  instance  through  the  prior  and  demand  distributions.  A  statistic  is  sufficient 
if  given  a  prior  distribution  on  the  parameter  t,  /(r),  the  posterior  distribution  on  the 
parameter  r,  /((T|5t),  is  a  function  of  the  sufficient  statistic  only,  and  not  the  individual 
sample  observations.  Therefore,  to  update  the  prior  distribution  of  a  parameter  to  a 
posterior  distribution  on  the  same  parameter,  only  the  value  of  the  sufficient  statistic 
needs  to  be  known.  DeGroot  [8]  discusses  sufficient  statistics  in  detail  and  provides 
numerous  examples.  The  initial  prior  distribution  is  /(t)  =  /o(t|  Jq).  The  current  prior, 
/t(r|Jt),  is  always  a  sufficient  statistic  that  is  easily  revised  using  Bayes’  formula 

tI7  1  =  T)ft{r\It) 

/o°° 


(2.1) 
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The  conditional  demand  distribution  depends  on  the  current  prior  through  the  equation 


f 


(2.2) 


We  now  present  our  general  model  formulation.  We  first  present  a  basic  model  and  show 
how  we  can  modify  it  to  account  for  positive  lead  time,  fixed  order  costs,  and  unobserved 
lost  sales.  When  lead  time  is  zero  with  backordering  and  no  fixed  costs,  the  control  action 
in  period  t  raises  the  inventory  level  to  a  value  y  with  an  expected  holding  and  backorder 
cost 


Lt{y\St)  =  [  h{y-  w)4>t{w\St)dw  +  [  p(w  -  y)<f)t{w\St)dw.  (2.3) 

Jo  Jy 

The  state  transition  equation  for  the  inventory  level  is  Xt^i  =  xt  +  at  —  wt^  and  the 
cost-to-go  function  is  then 

POO 

=  ini  {c{y  -  Xt)  +  Lt{y\St)  +  a  I  jf+y{y  -  w,St+i)4>t{w\St)dw}.  (2.4) 

Jq 

This  function  calculates  the  total  discounted  costs  from  time  t  to  the  end  of  the  horizon, 
T.  When  we  account  for  positive  lead  time,  we  augment  the  state  space  to  include 
the  outstanding  orders  at  time  t.  Define  the  vector  at,i  =  {at-i,at-i+i, . . .  ,ot-i),  then 
the  state  vector  at  time  t  is  {xt,St,dt,i).  The  inventory  transition  equation  is  xt+i  = 
xt  +  at-i  —  Wt,  and  the  order  transition  equation  is  dt+i,i  =  at,i  —  +  {oj}.  For  an 
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arbitrary  inventory  level  v,  define  the  single  period  expected  inventory  cost  function  as 

pv  roo 

Lt{v\St,l)=  h{v  -  wt,i)M'(i^t,i\Sul)dwt,i  +  p{wt,i  -  (2.5) 

Jo  Jv 

where  wt^i  =  function  accounts  for  the  positive  lead  time  in 

computing  the  expected  inventory  costs  for  the  inventory  level  at  time  t  +  L  The  cost- 
to-go  function  with  positive  lead  times  becomes 

i 

JT St,at,i)  =  M  {c{at)  +  Lt{xt  +  '^at-k\Sul)  + 

at>0 

A:=0 

roo 

Jt+i{^t  +  at-i-wt,St+i,at+i,i)<j)t{w\St)dw}.  (2.6) 
Jo 

This  problem  cam  be  simplified  by  formulating  it  in  terms  of  the  amount  of  inventory 
on-hand  and  on-order  at  the  beginning  of  t,  ut  =  xt  +  Yli=i  which  is  referred 
to  as  the  inventory  position.  The  inventory  position  follows  the  transition  equation 
ut+i  =  Ut  +  at  —  wt,  and  the  DP  recursion  becomes 

T 

Jt  (“f)  St)  =  i^f  {c(of)  4-  Lt{ut  +  at\St,l)  +  a  Jt+ii'^t  +  oj  —  wt,  St+i)^t{‘>u\St)dw}. 

at>0  Jq 

(2.7) 

Including  lead  time  will  significantly  increase  the  effect  of  uncertainty  in  the  problem 
due  to  the  uncertainty  of  demand  over  the  lead  time.  Adding  a  fixed  cost  to  the  problem 
only  requires  a  slight  modification  to  the  model  at  this  point.  The  cost-to-go  functions 
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becomes 


jJ (t6t,  St)  =  inf  {K  ■  l{at  >  1}  +  c{at)  +  Lt{ut  +  atlSt,  1)  + 

at  >0 


TOO 

^  /  Jl^i{y>t  +  (^t-'^uSt^i)(t)t{w\St)dw},  (2.8) 


where  1{*}  is  an  indicator  function.  The  final  addition  to  our  model  is  that  of  unob¬ 
served  lost  sales.  The  primary  distinction  from  this  additional  assumption  is  that  the 
measurement  process  is  less  precise.  When  stockouts  are  backordered,  the  demand, 
is  the  measurement  process  of  the  underlying  process  because  the  demands  are  directly 
observed.  We  assume  that  when  stockouts  are  lost  that  only  actual  sales  are  observed 
so  that  {zt}  where  zt  =  min{xt  +  at^i.wt}  is  the  measurement  process.  In  this  case, 
zt  is  the  actual  sales  in  period  t  as  opposed  to  the  demand  wt-  The  unobserved  lost 
sales  case  results  in  censored  observations  and  increases  the  value  of  on-hand  stock  as  a 
means  of  more  accurately  observing  demand.  In  lost  sales  cases,  the  penalty  cost  must 
also  account  for  lost  profit  per  unit  of  lost  sales.  The  information  vector  and  suiEcient 
statistic  may  be  more  difficult  to  calculate  due  to  the  unobserved  lost  sales.  This  leads 
to  the  final  form  for  the  cost-to-go  function  in  our  model. 


jJ {xu  St,  at,i)  =  inf  {iC  •  l{ot  >  1}  +  c(ot)  +  Lt{xt  +  at-i\St)  + 


at>0 


roo 

Jt+ii^t  +  at-l-Zt,St+i,at+i,i)4>tiz\St)dz}.  (2.9) 
Jo 
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Also,  the  cost-to-go  function  must  include  the  vector  of  orders,  at,i,  because  the  possibility 
of  lost  sales  precludes  the  simplification  from  using  the  inventory  position  as  a  state 
variable.  This  greatly  complicates  computation  for  the  lost  sales  case  with  positive  lead 
time. 

There  are  some  very  important  modeling  assumptions  that  arise  in  this  basic  formu¬ 
lation.  The  most  critical  assumptions  are  the  choice  of  an  a  priori  distribution,  /,  for  the 
unknown  parameter  r  and  the  choice  of  a  distribution  for  demand,  ^{w;  t).  These  choices 
are  particularly  important  when  r  is  continuous  because  they  affect  the  tractability  of  the 
problem.  If  the  a  priori  and  demand  densities  are  chosen  from  a  pair  of  conjugate  fami¬ 
lies,  then  the  Bayesian  calculation  of  the  posterior  distribution  of  t  is  greatly  simplified; 
otherwise,  this  calculation  requires  numerical  integration.  Furthermore,  the  cost-to-go 
function  in  Equation  (2.4)  has  two  state  variables,  xt  and  St-  An  even  more  selective 
choice  of  densities  can  lead  to  a  reduction  of  the  state  space  to  a  single  dimension  which 
further  improves  the  tractability  of  the  model.  The  drawback  to  these  assumptions  is 
that  they  may  lead  to  a  model  that  does  not  accurately  approximate  the  distribution 
for  the  actual  problem.  In  these  cases,  r  may  be  modeled  as  a  discrete  parameter  with 
finite  range,  the  calculation  of  the  posterior  distribution  is  simplified  and  does  not  re¬ 
quire  the  use  of  conjugate  families.  However,  there  are  no  known  results  that  reduce  the 
size  of  the  state  space  for  the  discrete  problem.  This  situation  leads  to  an  opportunity 
for  an  interesting  line  of  research  which  is  to  investigate  the  effectiveness  of  developing 
a  near-optimal  control  policy  for  a  discretized  approximation  of  the  true  prior  /. 
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There  are  also  some  interesting  and  useful  properties  which  are  known  for  this 
problem.  The  policy  structure  is  similar  to  that  of  the  fully  observed  problem  except 
that  it  is  dependent  on  the  current  prior,  Mr).  In  the  case  of  no  fixed  ordering  cost 
with  linear  purchase,  holding,  and  penalty  costs,  the  policy  is  a  state-dependent  base 
stock  control  policy,  5'(/t(r)),  see  Scarf  [33].  In  the  case  of  a  positive  fixed  order  cost  the 
policy  is  state-dependent  with  two  critical  numbers,  (s(/t),  S{ft)),  see  Lovejoy  [28].  These 
policy  structures  reduce  the  computational  requirements  and  are  an  intuitive  extension 
of  the  policies  for  the  fully  observed  problems.  Another  important  property  is  that  in 
the  case  of  back  ordering  the  optimal  policy  converges  to  the  maximum  likelihood  policy, 
i.e.,  the  MLE  of  t,  f,  is  assumed  to  be  the  known  value  of  t,  see  Scarf  [33].  This  means 
that  once  enough  information  is  gathered  about  the  value  of  t,  then  the  state-dependent 
policy  can  be  replaced  by  the  certainty  equivalent  policy  using  f  without  a  significant 
loss  of  optimality.  Thus,  an  interesting  research  question  is  raised,  which  is  to  determine 
when  this  policy  transition  should  take  place. 

2.3  Optimal  Control 

2.3.1  Uncensored  Observations 

Many  practitioners  assume,  although  sometimes  incorrectly,  that  their  demand  is 
observed  exactly.  When  this  is  the  case,  the  solution  to  the  problem  is  somewhat  sim¬ 
plified.  The  following  section  discusses  uncensored  observation  cases. 
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Basic  Properties 

Several  authors  consider  the  case  of  full  backordering.  Similar  results  would  hold  in 
lost  sales  cases  in  which  demand,  although  lost,  is  still  fully  observed.  We  first  discuss 
some  general  properties  of  the  problem  from  three  papers  by  Scarf  [33],  Karlin  [17],  and 
Iglehart  [15].  We  then  provide  results  from  Scarf  [34]  and  Azoury  [4]  that  are  not  only 
more  specific  in  nature,  but  also  provide  a  reduction  in  the  state  space  for  the  problem. 

Herbert  Scarf  [33]  was  a  pioneer  in  the  use  of  Bayesian  techniques  for  inventory 
control.  Scarf  assumes  that  holding,  penalty,  and  procurement  costs  are  all  linear.  There 
are  no  fixed  order  costs,  no  end  of  horizon  salvage  value,  lead  time  is  zero,  demand  is 
backordered,  and  futmre  costs  are  discounted.  He  also  assumed  that  the  demand  density 
was  in  the  exponential  class:  {<l>{w;r)  =  /3{T)e~^'^r{w)}  where  r{w)  =  0  for  u>  <  0.  The 
exponential  class  of  distributions  (including  Gamma,  Poisson,  and  negative  binomial)  is 
particularly  convenient  because  it  has  a  scalar  sufficient  statistic,  St  =  ujj- 

Scarf’s  first  key  result  is  that  optimal  policies  for  this  problem  are  defined  by  critical 
numbers,  Xt(S't),  i.e.,  base  stock  control.  Scarf  proves  that  xt{St)  is  the  unique  solution 
to  Equation  (2.4),  and  that  it  is  monotone  increasing  in  St-  Additionally,  when  demand 
is  sufficiently  small  in  t,  then  no  order  is  placed  in  the  following  period,  t  + 1.  Analogous 
results  hold  for  the  infinite  horizon  version. 

Since  critical  numbers  are  difficult  to  obtain  in  general,  a  very  important  result  is 
that  the  critical  numbers,  xt{St),  have  an  eisymptotic  relationship  to  the  critical  numbers 
in  the  case  that  the  demand  distribution  is  fully  known  with  mean  equal  to  St-  Define 
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this  critical  number  as  X{St),  then  xt{St)  ~  X{St)  +  Scarf  also  provides  the 

explicit  equation  for  a{St),  which  may  have  a  positive  or  negative  value.  This  asymptotic 
relationship  implies  that  the  Bayesian  stock  level  converges  to  the  maximum  likelihood 
policy.  So,  for  sufficiently  large  t,  the  Bayesian  stock  level  is  readily  approximated. 

Karlin  [17]  first  analyzes  a  “non-stationary”  problem  in  which  the  distribution  can 
vary  each  period.  Using  those  results,  he  then  considers  our  case  of  stationary  demand 
in  which  the  demand  is  not  fully  known.  Karlin  examines  two  cost  structures,  one  with 
linear  purchase  costs  and  another  with  convex  purchase  costs.  In  each  case,  holding  and 
penalty  costs  are  convex.  Karlin  asserts  that  similar  results  hold  when  there  are  positive 
lead  times.  He  also  allows  both  backordering  and  observable  lost  sales.  He  extends  the 
results  of  Scarf  (exponential  family)  to  include  the  range  family  and  provides  some  or¬ 
dering  properties.  As  in  Scarf,  there  is  an  a  priori  distribution  on  the  unknown  demand 
parameter.  Distributions  in  the  range  family  are  of  the  form  4>{w,  r)  =  'Y{T)q{w)ip{w;  r) 
where  ^(w;  r)  =  1  for  0  <  lo  <  t,  and  0  otherwise.  The  range  family  includes  all  continu¬ 
ous,  positive  random  variables  with  a  bounded  density  that  is  truncated  at  some  positive 
number  r,  e.g.,  uniform.  The  sufficient  statistic  for  this  family  is  St  =  maxi<i<t  lOj.  Kar¬ 
lin  demonstrates  two  stochastic  orderings  for  both  the  exponential  and  range  families. 
Karlin  shows  that  for  a  given  time,  t,  a  smaller  sufficient  statistic  will  imply  that  the 
conditional  demand  density  is  stochastically  smaller  than  the  density  associated  with  the 
larger  statistic:  If  St  <  S^,  then  0t(-|iS't)  is  stochastically  smaller  than  (written 
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^t{-|5't)  C  <^t(-|<S'Q).  For  a  fixed  value  of  a  suflBcient  statistic  {S  =  St+i  =  St),  the  condi¬ 
tional  demand  density  at  t  -I- 1  is  smaller  than  the  density  at  time  t:  (j)t+i (-[S')  <  ^t{-\S). 
With  linear  purchase  costs  and  other  costs  convex,  the  optimal  policy  is  a  single  critical 
number,  xt{St).  If  the  purchase  costs  are  generalized  to  be  strictly  convex  with  linear 
shortage  costs  and  convex  holding  costs,  the  policy  is  a  function  of  two  numbers.  The 
policy  is  of  the  (s,  S)  type  and  is  to  order  up  to  y{x-,  ^i,  </>2, . . . )  if  x  <  x(^i,  ); 

otherwise,  do  not  order.  This  leads  to  the  result  that  for  a  one-period  model  in  which 
there  is  a  sufficient  statistic  St  or  S't  where  {St  <  S't),  the  order-up-to  quantities  for  the 
two  respective  cost  structures  are  such  that  xi(S't)  <  xi(5'j)  and  yi(x|5t)  <  yi(x|5^)  for 
all  X  >  0.  Then,  by  induction,  where  again  {St  <  S't),  the  initial  order-up-to  quantities 
for  the  T  period  problem  are  such  that  XT{St)  <  XT{S't)  and  yj.(x|5t)  <  yx{x\S't)  for  all 
X  >  0. 

Iglehart  [15]  extends  the  work  of  both  Scarf  and  Karlin.  He  extends  Scarf  by  in¬ 
cluding  the  range  family  as  well  as  the  exponential  family  in  his  results.  The  shortage 
and  penalty  costs  are  convex  while  procurement  costs  remain  linear.  Iglehart  assumes 
full  backordering  and  zero  lead  time.  However,  similar  results  still  hold  when  either 
assumption  is  relaxed.  As  in  Scarf  and  Kaxlin,  there  is  an  unknown  parameter  for  the 
demand  which  has  a  known  a  priori  distribution.  Iglehart  also  presents  a  more  com¬ 
plete  set  of  theoretical  properties  and  proofs  than  Karlin.  He  first  demonstrates  that 
the  optimal  policy  in  Equation  (2.4)  is  represented  by  a  single  critical  number,  XT{St)- 
The  cost  function,  J^{xt,St),  is  convex  with  respect  to  x.  He  then  shows  that  for 
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a  fixed  St  that  xriSt)  <  XT+i{St)  and  that  xriSt)  <  x(St)  where  x(St)  is  smallest 
root  of  the  equation  c(l  —  o:)  +  L'lyjSt)  =  0.  Also,  limy  =  J(a:i5t).  He 

presents  more  complete  results  than  Karlin  about  how  the  critical  numbers  vary  with 
changes  in  the  information.  He  shows  that  for  St  <  S't,  then  XT{St)  <  xxiS't)  and 
■^J^{x\St)  >  -^J^ixlSl).  That  is,  a  larger  sufiicient  statistic  at  time  t  will  lead  to  an 
equal  or  larger  base  stock  level  for  the  T-period  horizon  problem.  Also,  the  slope  of  the 
cost  function  will  be  larger  at  any  point  x  for  the  fimction  with  the  smaller  sufficient 
statistic.  He  proves  the  following  results  for  all  fixed  values  of  S  and  x.  The  critical 
number  XT{S,t  +  l)  <XT{S,t)  while  ^J'^{x\S,t  +  1)  >  J^{x\S,t).  Iglehart  shows  the 
asymptotic  behaviors  of  the  conditional  demand  when  the  sufficient  statistics  are  fixed 
for  large  t  for  both  exponential  and  range  cases.  For  example,  if  the  maximum  likelihood 
estimator  of  t  is  ft,  and  if  /(ft)  >  0  and  /(t)  is  continuous  in  the  neighborhood  of 
ft,  then  hmt->.oo  ^(u;|5t)  =  <f>{w\ft).  Scarf  showed  that  limt_>oo  *^(2;,  ^t)  =  J{x,ft)  and 
\imt^oox{St)  =  x{ft).  He  concludes  by  showing  the  following  asymptotic  property.  If 
fi^t)  >  0  and  continuous,  then  x{St)  ~  x(ft)  +  o(ft)  as  t  ^  00  where  a(ff)  is  defined 
appropriately  for  both  the  exponential  and  range  families. 

State  Space  Reduction 

We  now  focus  attention  on  results  that  greatly  simplify  the  calculation  of  optimal 
policies.  In  a  subsequent  paper  [34],  Scarf  shows  that  in  some  instances  the  critical  num¬ 
ber  can  be  determined  as  a  function  of  one  variable.  He  first  assumes  that  procurement. 
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holding,  and  penalty  costs  are  all  linear.  Lead  time  is  zero,  but  he  notes  that  his  pro¬ 
cedures  can  be  adapted  for  use  if  there  is  positive  leaxi  time  and  backordering.  He  also 


assumes 


that  the  demand' distribution  is  in  the  gamma  family:  <P{w;t)  =  - , 


where  o  is  a  known  constant  and  r  is  an  unknown  parameter.  The  a  priori  distribution 
is  also  gamma  with  known  parameters,  A  and  b,  so  that  /(t)  =  this  case 

the  sufficient  statistic  is  St  =  He  first  demonstrates  the  relationship  that  re¬ 

duces  the  problem  to  one  of  a  single  state  variable.  The  conditional  demand  distribution 
can  be  reformulated  eis 


W 


St  +  Xj' 


(2.10) 


where  = 


T{ta  +  b)  w 


a-l 


T{a)T[{t-l)a  +  b)] 


(2.11) 


He  then  demonstrates  that  the  cost  function  can  be  computed  as  a  scalar  multiple  of  a 
related  single  state  variable  problem,  i.e.,  Jt{x,  St)  =  {St  ■+  -^)>^t(5j^)» 

where  Jt{x)  =  ^n  |c(j/  -  x)  +  Lt{y)  +  «^t+i  (1  +  z)<pt{z)dz^  .  (2.12) 


He  also  shows  that  the  critical  numbers  are  proportional  to  X  +  St-  Because  of  the 


state  spaxje  reduction  in  this  special  case,  the  solution  to  the  inventory  problem  with 
incomplete  information  is  no  more  difficult  than  a  problem  with  complete  information. 
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This  reduction  in  the  state  space  should  be  very  attractive  to  practitioners  facing  these 
types  of  problems. 

Azoury  [4]  extended  Scaxf’s  work  by  showing  more  general  conditions  for  a  reduc¬ 
tion  in  the  state  space  of  the  Bayesian  model.  Azoury  assumed  a  finite  horizon  with 
backordering  and  that  the  ordering,  holding,  and  shortage  costs  are  linear  and  and  lead 
time  is  zero.  Azoury  considers  both  depletive  and  nondepletive  (repairable)  inventory 
models.  She  chooses  a  series  of  demand  distributions  that  have  a  prior  which  comes 
from  a  conjugate  family.  This  ensures  that  the  posterior  distribution  on  the  unknown 
parameter  is  in  the  same  family  as  the  prior.  See  DeGroot  [8]  for  a  detailed  description 
of  conjugate  families. 

We  will  examine  results  of  the  depletive  model.  Once  again  the  model  is  defined  as 
in  Equation  (2.4).  The  two  conditions  that  allow  the  state  space  reduction  are: 

1.  There  exists  a  set  of  functions  qt{St)  such  that  ^t(u;|5t)  =  (glpd) 

where  is  a  probability  density  function  of  scaled  demand  that  depends  only 
on  t. 

2.  qt+-,iSt+i)=qt{St)Ut+i{^^)  for  some  continuous  real  valued  function  Ut+i  such 
that 


/o°°  Ut+iiz)t()tiz)dz  <  oo. 
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Demand 

Prior 

Suff.  Stat. 

xt{St) 

Uniform 

Pareto 

max  Wi 

7tmax(5t,R) 

f  1/r  if  0  <  m  <  T 
0{w\  r)  =  < 

1^0  otherwise. 

g{T)  =  aRVT“+i 

T  >  R;  a,  R  known 

Weibull 

Gamma 

4>{w,  t)  = 

9\V  -  — r(S) — 

w>Q,k  known 

r  >  0;  a,  6  >  0 

Periodic/Gamma 

Gamma 

Y^iwilh) 

lth{h  +  St) 

=  ktZt  kf  is  periodic  index 

9{V  —  r{a) 

1—1 

2  >  0;  A,t  >  0 

T  >  0;  a,  6,  >  0 

Table  2.2:  Distributions  with  State  Space  Reductions 
The  equation  of  the  single  variable  is 


Vt{x)  =  min{c(?/  -  x)  +  Ht{y)  +  Ut+i{^)'^t{z)dz},  (2.13) 

where  Ht{y)  =  h ^^{y  —  z)xl}t{z)dz  +p  J^{z  —  y)‘ipt{z)dz  for  y  >  0.  The  function  14(‘) 
is  convex,  hence  a  base  stock  policy  is  optimal.  Let  jt  achieve  the  minimum  of  Vt{x)  in 
(2.13).  Azoury  then  proves  for  the  original  problem  that  xt{St)  =  7t9t(5't).  Table  2.2 
shows  three  sets  of  distributions  that  satisfy  the  conditions.  Note  that  for  kt  =  1,  the 
third  example  is  the  result  due  to  Scarf  [34].  The  last  colrunn  shows  the  optimal  order- 
up-to  level  for  the  finite  horizon  model  and  is  a  function  of  the  sufficient  statistic.  The 
critical  level  is  always  proportional  to  7*,  the  value  of  y  that  minimizes  Equation  (2.13). 
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Azoury’s  results  for  the  nondepletive  models  are  analogous  to  those  of  the  depletive 
models.  Additionally,  the  results  of  the  depletive  model  also  hold  for  positive  lead  time. 
When  Azoury  or  Scarf’s  assumptions  hold,  managers  can  often  solve  partial  information 
problems  with  relative  ease. 

Scarf  and  Azoury  showed  how  under  certain  conditions  these  problems  can  be  re¬ 
duced  to  single  dimensional  state  space.  Lovejoy  [25]  adds  two  more  assumptions  to 
those  of  Azoury  and  shows  under  these  additional  assumptions  that  policies  can  be  re¬ 
duced  to  a  zero  dimensional  state  space  and  solved  by  a  simple  critical  fractile  policy 
For  problems  that  do  not  satisfy  these  additional  conditions,  Lovejoy  shows  bounds  on 
the  value  loss  of  these  critical  fractile  policies  versus  an  optimal  policy,  Lovejoy  examines 
an  N-horizon,  discounted  problem  with  zero  lead  time  and  linear  costs.  Excess  demand 
is  backordered  although  a  certain  fraction  of  backorders  become  observed  lost  sales.  He 
specifically  addresses  exponential  smoothing  and  Bayesian  cases  along  with  numerical 
examples.  The  cost  is:  J  =  +  oFgxt+\  where  a  is  the  salvage 

value  and  a  is  the  action  that  takes  the  inventory  level  to  x.  This  problem  can  be  solved 
by  dynamic  programming  with  the  state  space  consisting  of  the  current  inventory  and 
the  sujfficient  statistic.  The  additional  assumptions  are 

1.  For  f  =  1, 2, . . ,  , T,  the  current  cost  function  Ct{x^ a,  ty)  =  Ltx  +  Kt{a^  w)  for  some 
constant  Lt  and  function  Kt^  and  with  Lt+i  =  cr. 

2.  The  transition  equation  for  the  inventory  level  is  independent  of  the  current  stock 
level,  i.e.,  xt+i  =g^(x,a,^^;)  =  gt{a,w). 
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If  these  assumptions  hold  in  addition  to  Azoury’s  conditions  then, 

Gt{y)  =  /o°° Kt{y,0  +  aLt+igt{y,OdFtiO,  t  =  1, 2, . . .  ,T.  Let  be  the  global  min- 
imizer  of  Gt  which  can  be  directly  calculated  from  the  problem  data.  The  inventory 
problem  can  be  solved  in  a  series  of  T  one-step  minimizations  of  Gt-  The  optimal  action 
at  time  t  is  Si{xt,St)  =  qt{St)yt-  If  excess  inventory  can  be  disposed  of  at  cost,  then 
Lovejoy’s  conditions  are  satisfied  and  the  optimal  policy  is  the  solution  of 


Ftiyt)  =  = 


p  —  c(l  —  a4>) 
h+p  +  a{(l>  -  7) 


(2.14) 


where  7  inventory  deteriorates  each  period  and  (j)  backorders  are  lost  each  period.  Love- 
joy  considers  this  same  policy  when  disposal  is  not  allowed,  and  it  only  meets  those 
assumptions  proposed  by  Azoury.  He  then  shows  the  upper  bound  on  the  loss  relative 
to  optimal  for  using  this  simpler  policy.  He  shows  when  it  is  feasible  to  hold  onto  stock 
rather  than  disposing  it  and  the  value  of  using  a  nondisposal  policy. 

Lovejoy  demonstrates  these  models  with  examples  using  either  exponential  smooth¬ 
ing  or  Bayesian  updating  and  makes  several  qualitative  conclusions.  For  example,  myopic 
policies  perform  better  when  perishability  is  high.  As  expected,  myopic  policies  perform 
better  as  the  precision  of  the  information  on  the  unknown  parameter  increases.  The 
conditions  of  Lovejoy  are  similar  to  those  we  will  see  later  in  Lariviere  and  Porteus  [21] 
because  they  assume  stock  deteriorates  at  the  end  of  each  cycle.  They  are  both,  in  effect, 
eliminating  the  incoming  stock  level  from  the  state  space. 
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2.3.2  Censored  Observations 

The  previous  papers  all  assumed  that  the  demand  is  always  completely  observed. 
However,  when  stockouts  are  lost,  the  demand  is  often  only  partially  observed.  That  is, 
if  there  is  no  inventory  left  at  the  end  of  a  period,  the  decision-maker  may  not  know  the 
amount  of  lost  sales,  in  which  case  the  demand  process  is  only  partially  observed  through 
the  sales.  This  situation  generates  additional  uncertainty  about  the  demand  process  and 
probably  occurs  frequently  in  practice.  It  also  provides  an  additional  incentive  to  hold 
stock  in  order  to  gather  more  accmate  demand  information. 

Although  they  do  not  study  an  adaptive  inventory  control  model,  Agrawal  and 
Smith  [1]  and  Nahmias  [30]  consider  the  problem  with  unobserved  lost  sales  and  make 
some  observations  that  axe  beneficial  to  our  problem.  Agrawal  and  Smith  argue  that  the 
negative  binomial  family  often  fits  actual  demand  data  well  when  the  variance  is  high 
as  in  the  case  with  many  retailing  cases.  They  describe  how  to  fit  a  negative  binomial 
distribution  to  unobserved  lost  sales  data  and  demonstrate  the  goodness-of-fit  for  some 
actual  sales  data.  Their  technique,  although  not  optimal  for  the  Bayesian  problem, 
could  be  used  to  update  parameter  estimates.  Nahmias  [30]  also  describes  a  parameter 
estimation  procedure  for  unobserved  lost  sales  using  the  normal  distribution  family.  He 
develops  a  simple  estimation  procedure  and  demonstrates  its  accuracy  compared  to  the 
maximum  likelihood  estimates.  Again,  managers  should  consider  these  procedures  to 
update  parameter  estimates  when  there  are  censored  observations. 
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Lariviere  and  Porteus  [21]  describe  an  optimal  solution  to  a  finite  horizon  problem 
that  uses  Bayesian  techniques  for  the  unobserved  lost  sales  problem.  This  paper  di¬ 
rectly  extends  Azoxury’s  results  to  the  case  where  demand  observations  are  censored  due 
to  unobserved  lost  sales.  All  costs  and  revenue  are  linear  and  lead  time  is  zero.  The 
identification  of  conjugate  distributions  is  more  difficult  in  this  case.  The  authors  model 
both  a  single  retailer  as  well  as  a  multiple  market  problem.  The  retailer’s  objective  is  to 
maximize  his  profits  over  a  finite  horizon.  They  assume  that  the  retailer’s  excess  inven¬ 
tory  each  period  is  lost.  For  the  retailer,  this  decouples  inventory  levels  between  periods, 
but  the  state  of  information  from  the  previous  periods  still  couples  the  periods.  The 
perishability  assumption  simplifies  the  problem  significantly,  but  also  limits  application 
of  the  results.  The  retailers  use  a  base  stock  policy  that  depends  on  the  current  infor¬ 
mation.  The  retailer  can  learn  more  about  the  true  demand  process  by  holding  stocks 
at  a  higher  level  initially  to  more  quickly  establish  a  better  estimate  of  the  true  demand. 
A  true  measure  of  demand  occurs  only  when  stock  is  left  over  at  the  end  of  the  period. 
Braden  and  Preimer  [6]  define  a  family  of  “newsvendor”  distributions  that  have  a  fixed 
dimension  statistic  even  with  censored  observations.  These  distributions  are  of  the  form 
^{w;t)  =  1  —  .  The  Weibull  distribution  {(p{w,k,T)  =  ;w  >  0} 

is  the  only  “newsvendor”  distribution  that  fits  Azoury’s  requirements  for  a  reduction  in 
the  state  space  and  is  therefore  the  distribution  used  by  the  authors  {k  known  and  r 
unknown).  The  appropriate  prior  for  r  at  time  t  in  this  case  is  T{at,bt).  The  parameters 
are  updated  each  period  by  the  number  of  censored,  me,  and  uncensor  ed/exact,  rug, 
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observations  along  with  the  last  sales  observation,  wt-i-  Both  parameters  are  a  measure 
of  market  size,  but  only  the  shape  parameter  is  a  measure  of  the  precision  of  information 
on  the  true  demand  distribution  since  the  coefiicient  of  variation  for  the  gamma  prior  is 
y/l/at-  The  parameters  at  time  t  are 

t 

at  =  ai+rrie  =  hi  +  ^  d{wi)  where  d{wi)  =  wf  for  a  fixed  k. 

i=l 

Notice  that  the  shape  parameter  at  only  increases  with  exact  observations.  This  fact 
is  vital  to  the  development  of  the  policies.  The  sufficient  statistic,  St,  is  the  vector 
of  current  parameters,  {at,  bt),  for  the  prior.  They  show  that,  as  in  Azoury,  the  scale 
parameter  can  be  eliminated  firom  the  analysis  such  that  Jt{at,bt)  =  q{bt)Vt{at).  The 
simplified,  single  dimensional  system  comes  from  normalizing  the  initial  inventory  level 
and  the  expected  demand.  As  in  Azoury,  the  optimal  order-up-to  level  is  Xt  =  q{bt)'it{0't), 
where  7t(at)  solves  the  reduced  state  system. 

Lariviere  and  Porteus  demonstrate  several  interesting  results  for  managers  to  con¬ 
sider.  If  there  is  no  penalty  cost,  the  retailer  will  always  prefer  stockouts.  When  stockout 
penalties  exist,  the  retailer  will  usually  prefer  excess  inventory  to  reduce  the  level  of  un¬ 
certainty.  Therefore,  retailers  should  often  stock  more  inventory  than  is  necessary  to 
maximize  current  profits  in  order  to  obtain  more  precise  information  on  demand.  For  a 
given  purchase  cost,  the  optimal  discounted  expected  return  per  unit  of  mean  demand 
is  strictly  increasing  in  the  number  of  exact  observations.  When  stockout  penalties  axe 
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small,  a  product  that  should  be  stocked  will  never  be  dropped  despite  poor  sales.  More¬ 
over,  although  not  necessarily  intuitive,  a  product  that  is  successful  and  stocks  out  might 
be  dropped  as  a  result.  This  is  because  the  inability  to  increase  the  precision  of  infor¬ 
mation  may  lead  to  a  situation  where  the  risk  of  carrying  too  much  inventory  at  a  point 
closer  to  the  end  of  the  horizon  is  too  high  to  stock  the  product. 

Lariviere  and  Porteus  extend  the  model  to  a  multiple  market  problem  by  assuming 
there  are  j  “independent”  markets,  the  largest  of  which  is  noted  as  the  primary  mar¬ 
ket.  The  demand  distributions  are  all  Weibull,  and  the  primary  market  has  an  unknown 
parameter  with  a  gamma  prior.  The  demand  in  the  other  markets  are  proportional  to 
the  demand  in  the  primary  market.  Products  cannot  be  dropped  and  the  markets  dif¬ 
fer  only  in  size.  The  decision  maker  has  at  least  three  stocking  options.  The  “Naive” 
option  treats  all  markets  completely  independently.  A  “Pooled  Information”  approach 
uses  knowledge  from  all  the  markets  to  update  the  unknown  parameter  but  the  stock¬ 
ing  decisions  remain  independent.  The  third  “Joint  Optimization”  approach  considers 
all  interactions  between  markets  in  setting  stocking  levels.  There  are  some  interesting 
interactions.  When  the  shape  parameter  is  very  low  or  high,  there  is  a  complementary 
effect  that  causes  the  overall  stocking  levels  to  be  higher  than  when  levels  are  considered 
independently.  This  is  because  each  market  has  an  incentive  to  stock  higher  levels  to 
provide  more  information  for  other  markets.  On  the  other  hand,  in  many  (but  not  all) 
problems  with  intermediate  values  for  the  shape  parameter,  a  substitution  effect  causes 
lower  stocking  levels  in  the  joint  problem.  Since  a  market  receives  information  from  other 
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markets  it  has  an  incentive  to  stock  at  lower  levels.  Returns  are  higher  when  information 
is  pooled  as  opposed  to  naive  policies.  If  the  shape  paxameter  indicates  any  moderate  or 
higher  level  of  precision,  then  the  joint  optimization  only  performs  nearly  the  same  as 
the  pooled  information  case.  The  authors  also  provide  a  persuasive  argument  that  ex¬ 
perimenting  to  gain  information  is  better  done  in  a  smaller  maxket  than  a  larger  market. 
The  reasoning  is  that  stocking  one  additional  unit  in  a  smaller  market  is  more  likely  to 
lead  to  an  exact  observation  than  in  the  larger  market.  Due  to  the  proportionality  of 
demand,  stocking  one  unit  in  a  smaller  market  is  equivalent  to  stocking  one  times  the 
proportionality  constant  in  the  larger  market.  Finally,  observations  in  a  smaller  market 
are  at  least  as  important  as  in  the  larger  market. 

2.4  Suboptimal  Control 

2.4.1  Continuous  Parameter 

Often  times  managers  must  make  a  series  of  decisions  before  any  demand  observa¬ 
tions  axe  available  to  update  unknown  parameters.  Kaplan  [16]  considers  a  spare  parts 
problem  for  a  new  system.  In  this  case,  it  is  often  necessary  to  make  some  purchase 
decisions  before  fielding  the  new  system,  and  thus,  before  actual  demand  is  observed. 
In  fact,  several  orders  may  be  placed  before  fielding  if  the  lead  time  is  long,  and  once 
demands  occiu,  the  information  is  used  to  adjust  inventory  levels.  Kaplan  also  assmnes  a 
fixed  order  cost  and  a  periodic  (s,  S)  control  policy.  He  does  not  state  restrictions  on  the 
holding  and  penalty  costs,  but  we  assume  the  results  hold  if  they  are  convex.  There  is  full 
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backordering  and  known  constant  lead  times  for  procurement.  He  assumes  the  demand 
distribution  is  known  and  has  an  unknown  parameter  with  an  a  priori  distribution.  He 
states  that  practicality  limits  choice  of  the  prior  to  one  from  a  conjugate  family.  An 
important  issue  for  this  problem  is  to  determine  how  prefielding  decisions  change  given 
that  demand  estimates  change  once  fielding  takes  place.  His  dynamic  programming 
formulation  is  the  same  as  Equation  (2.4)  where  St  =  ^*=1  Wi- 

L{y\St)  is  the  expected  present  value  of  backorder  and  holding  costs  in  period  {t  + 
L  +  1),  since  inventory  ordered  at  time  t  does  not  arrive  until  t  +  L.  Kaplan  assumes 
that  the  Bayesian  updates  will  cease  at  some  designated  time  and  uses  a  heuristic  to 
estimate  the  cost  from  that  time  until  the  end  of  the  horizon.  As  long  as  this  time  is 
chosen  carefully,  it  should  not  adversely  affect  the  solution.  Kaplan  uses  an  (s,  S)  policy, 
although  he  does  not  prove  its  optimality.  See  Appendix  for  a  sketch  of  this  proof.  He 
computes  an  {s,S)  policy  using  the  following  steps.  Let  y*{x,St)  be  the  optimal  order- 
up-to  level  if  the  inventory  level  is  x  with  sufficient  statistic  St-  This  is  the  value  which 
minimizes  Equation  (2.4). 

1.  Find  y*(0, 5't)  and  set  S  =  y*{0,St) 

2.  Find  smallest  x  such  that  y*{x,St)  =  x  and  set  s  =  re  —  1. 

Kaplan  runs  some  experiments  for  Poisson  demand  with  a  gamma  prior.  A  key  part  of 
his  analysis  was  to  examine  how  uncertainty  about  the  unknown  parameter  affected  the 
results.  His  results  indicate  that  it  is  typically  better  to  order  less  and  wait  for  more 
information  after  the  item  has  been  fielded.  The  exact  effect  of  uncertainty  depends 
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on  backorder  costs,  degree  of  uncertainty,  and  the  time  until  fielding.  There  exists 
a  natural  tension  between  having  lower  reorder  points  until  more  information  becomes 
available  and  setting  higher  reorder  points  to  protect  against  shortages.  Kaplan  provides 
a  heuristic  which  ignores  the  imcertainty  in  the  prior  that  can  be  used  as  a  prefielding 
policy.  He  then  evaluates  the  increase  in  the  lifetime  costs  of  using  this  heuristic  at  two 
levels  of  uncertainty.  Although  this  hemristic  ignores  the  uncertainty  in  the  system,  it 
proved  useful  when  backorder  costs  were  relatively  low. 

Karmarkar  [18]  argues  that  inventory  problems  are  often  concerned  with  the  deter¬ 
mination  of  a  firactile  of  the  demand  distribution,  that  is,  the  service  level.  Therefore, 
it  is  often  sufiicient  to  develop  an  accurate  approximation  of  the  right  tail  of  the  de¬ 
mand  distribution  and  to  disregard  the  rest  of  the  distribution.  Karmarkax  provides  an 
approach  to  approximate  the  tail  distribution  of  demand.  This  procedure  may  be  more 
efiective  than  estimating  the  mean  of  a  distribution  and  then  inferring  information  about 
the  tail  fi:om  a  hypothesized  distribution.  It  may  be  more  accurate  to  estimate  the  tail 
of  the  distribution  from  the  beginning,  especially  when  the  distribution  is  difficult  to 
approximate,  e.g.,  bimodal  distributions.  His  approach  uses  Bayesian  analysis  as  well 
as  hemistic  smoothing  methods.  He  demonstrates  that  his  technique  outperforms  the 
Normal  approximation  for  certain  distributions,  is  simple,  and  uses  intuitive  statistics, 
although  the  method  has  some  inherent  bias.  The  approach  assumes  that  the  overall 
demand  is  a  mixture  of  2  probability  density  functions  as  shown  in  Figure  (2.2). 
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Here,  the  demand  density  function  is: 


fiw)  = 


I  Pdiw) 


0  <W  <Wo 


w>Wo 


(2.15) 


where  g{w)dw  =  1.  Note  that  p  is  the  probability  that  to  is  <  tOo-  The  distribution 
to  the  left  of  Wq  is  of  no  concern  in  determining  the  stock  level.  Kaxmarkar  suggests 
that  Wo  should  be  near  the  80%  fractile.  Above  the  set  point  Wq^  the  tail  distribution  is 
a  translated  exponential  which  accurately  approximates  the  tail  of  many  distributions. 
Given  that  Sl  is  the  desired  service  level,  with  known  parameters  Wqj  A,  and  p,  the 
order-up-to  level  is 


X  =  Wo-\-  -rln 


I-S'l 


(2.16) 
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Since  p  and  A  are  not  generally  known,  Karmarkar  provides  two  estimates  of  this  order- 
up-to  level.  The  first  he  calls  an  exact  estimator  which  is  derived  from  a  Bayesian 
analysis  based  on  the  distribution  of  demand,  unconditional  on  p  and  A.  It  accounts  for 
the  extra  uncertainty  that  is  present  in  the  system  due  to  estimating  p  and  A.  The  exact 
estimator  is 


Xl  =  Wo  +  Wg 


( 


1-p 

1-Sl 


_  1 


where 


(2.17) 


there  are  t  total  observations,  ni  is  the  number  of  points  less  than  or  equal  to  rig 
is  the  number  of  points  greater  than  Wo,  and  Wg  =  E[w\w  >  Wq]-  The  derivation 
of  the  exact  estimator  assumes  a  prior  distribution  on  both  p  (beta  with  parameters 
Wg  and  rig)  and  A  (gamma  with  parameters  ni  and  t).  An  approximate  estimator  is 
given  by  estimating  p  and  A  and  using  Equation  (2.16).  This  approach  is  a  type  of 
certainty  equivalent  control  that  assumes  p  and  A  are  exact.  Karmarkar  tests  these 
estimators  against  a  standard  Normal  distribution  using  both  MAD  and  variance  for 
four  distributions  (Normal,  exponential.  Erlang,  and  Normal  with  zero  spike).  The 
mixture  estimates  either  outperform  the  standard  approaches  or  perform  satisfactorily 
for  these  distributions.  The  approximate  estimator  always  provides  a  critical  fractile 
estimate  that  is  below  the  exact  estimate  from  Equation  (2.17). 
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2.4.2  Discrete  Parameter 

Lovejoy  [28]  models  the  problem  with  a  discrete  number  of  possible  demand  distri¬ 
butions.  He  computes  bounds  on  the  optimal  policy,  and  he  shows  how  to  iteratively 
tighten  both  the  upper  and  lower  bounds  on  the  optimal  policy.  He  also  shows  how  to 
generate  a  suboptimal  policy  in  the  process  of  computing  these  bounds.  His  model  has 
an  infinite  horizon  and  allows  fixed  ordering  costs.  Holding  and  shortage  costs  are  linear 
and  lead  time  is  zero.  Excess  demand  is  lost,  and  actual  demand  may  or  may  not  be 
observed  exactly. 

The  problem  is  modeled  as  a  composite-state  Partially  Observed  Markov  Decision 
Process  (POMDP).  Some  states  of  the  system  are  known  with  certainty  while  others 
are  only  partially  observed.  That  is,  the  inventory  level  is  known  with  certainty,  but  the 
demand  distribution  in  effect,  t  £  {1, . . .  ,n},  is  unknown.  The  objective  is  to  maximize 
the  expected  discounted  reward  over  the  infinite  horizon.  The  unknown  parameter,  r, 
is  the  demand  distribution  that  is  in  effect.  The  prior  for  r  is  represented  by  tt,  and 
define  the  set  H  =  {tt  £  i?”  :  ttj  >  0,  ^"=1  ttj  =  1}.  Lovejoy  uses  properties  of  POMDPs 
developed  by  Smallwood  and  Sondik  [35]  and  extends  suboptimal  techniques  used  by 
Van  Hee  [41]  to  suboptimally  solve  this  POMDP  problem.  If  a  partially  observable  state 
becomes  known  with  certainty,  the  problem  is  then  fully  observed  and  assumed  solvable 
by  conventional  dynamic  procedures.  Van  Hee  exploits  this  fact  to  show  how  upper  and 
lower  bounds  can  be  developed  by  solving  fully  observed  problems. 
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Lovejoy  develops  parallel  procedures  that  are  more  powerful  because  they  use  ap¬ 
proximate  value  functions  from  “action  invariant  sets”  to  set  bounds  and  adjust  them 
iteratively.  The  use  of  these  sets  gives  a  larger  class  of  cost  functions  to  approximate 
the  optimal  policy.  Because  it  is  well  known  that  the  maximum  reward  over  the  infi¬ 
nite  horizon  will  come  firom  a  stationary  Markov  policy,  let  J(x,  tt)  represent  the  to¬ 
tal  discounted  expected  reward  function  for  an  inventory  level  x  and  prior  tt,  that 
is  J{x,n)  =  E[Y^^-^a^~^r{xt,T^S{It))]  where  the  true  demand  distribution  is  t,  the 
action  is  S(It),  the  discount  rate  a,  and  r(-)  is  the  reward.  Define  the  value  func¬ 
tion  C(x,Tr,a,  J)  =  +  E{aJ{xt+i,nt+i)}.  Lovejoy  defines  a  mapping 

HsJ{x,'n)  =  C{x,‘K,6{x,Tr),J)  as  the  expected  current  reward  plus  the  future  reward 
using  value  fimction  J  and  a  feasible  policy,  6.  Finally,  let  H  J {x,  tt)  be  the  maximum 
present  and  future  reward  over  all  possible  actions,  a,  at  the  current  stage  and  let  J* 
be  the  optimal  value  function.  This  implies  that  a  specific  policy  5  is  optimal  if  and 
only  if  HgJ*  —  HJ*.  Although  it  is  impractical  to  do  so  since  the  state  space  in  11  is 
countably  infinite,  the  problem  could  theoretically  be  solved  by  dynamic  programming. 
The  following  key  relationship  is  used  to  generate  bounds  and  can  be  easily  verified  (let 
a  caret  ^  denote  a  fully  observed  problem  and  ej  is  a  unit  vector  with  a  1  in  the  ith 
position) 


n  n 

Jsix,ir)  =  ^niJs{x,ei)  =  '^'iriJ.^^ix). 

i=l 


(2.18) 
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Van  Hee  showed  that  a  lower  bound,  can  be  determined  by  solving  the 

n  fully  observed  problems.  Let  'ys{x)  =  be  an  n-vector  representing  the 

rewards  of  policy  6  and  value  function  J.  Each  component  of  the  vector  represents  the 
reward  if  the  true  demand  distribution  is  i.  Then,  Ji{x,n)  =  max.ygr(®){7  •  Van  Hee 
shows  that  a  one-step  dynamic  recmrsion  using  Ji  results  in  reward  HJi  and  the  following 
bounds  hold:  Ji  <  HJi  <  J*.  The  upper  bound  is  also  found  using  by  solving  the  n 
fully  observed  problems  where  Ju(a:,7r)  =  nJ*{x,ei).  Again,  a  one  step  recursion 
gives  a  reward  of  H  Ju  and  the  following  bounds  are  obtained: 

Ji  <  HJi  <J*<  HJu  <  Ju  (2.19) 

These  bounds  can  be  iteratively  tightened  in  theory  by  additional  dynamic  programming 
recmrsions  since  there  are  a  finite  number  of  states.  However,  the  state  space  grows 
exponentially  making  the  approach  practically  intractable. 

Lovejoy  first  extends  the  above  results  to  show  that  a  collection  of  invariant  sets 
('5'(a:))  can  be  used  to  generate  approximate  value  functions.  These  functions  can  be 
calculated  at  any  point  over  the  entire  state  space  {X  x  H)  rather  than  at  the  extreme 
points,  ej.  A  collection  of  sets  is  invariant  if  the  associated  policies  (actions)  are  a  function 
only  of  the  observed  state.  The  functions  that  are  generated  from  these  invariant  sets 
preserve  the  bounds  shown  in  Equation  (2.19).  Given  a  collection  of  invariant  sets,  ^(r), 
a  lower  bound  Ji  =  •  tt)  where  the  n-vector  ^  €  'i'(x).  The  lower  bound  is 
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found  analogously  to  Van  Hee,  although  now  the  collection  of  invariant  sets  may  include 
vectors,  tp,  generated  at  points  other  than  the  extreme  points,  {ci}.  The  upper  bound 
is  found  in  the  same  manner  as  in  Van  Hee  (Ju(x, tt)  =  'KiV*{x,ei)).  Once  again, 

a  single  dynamic  recursion  using  Ji  and  Ju  can  improve  these  bounds.  Boimds  are  tight 
near  the  extreme  points  as  well  as  at  points  where  observations  are  highly  informative 
about  the  underlying  demand  distribution.  They  may  also  be  tight  when  rewards  are  not 
observed  such  as  in  the  case  of  some  lost  sales  problems  when  only  sales,  not  demand, 
is  observed.  Since  this  technique  should  be  used  only  when  the  partially  observed  states 
have  a  significant  effect  on  the  rewards  and  transitions,  a  single  dynamic  programming 
recursion  (observation)  should  make  a  significant  impact  on  the  value  functions. 

Lovejoy  demonstrates  his  technique  with  an  inventory  example.  The  optimal  policy 
has  an  (s,  5)  structure  that  is  a  function  of  the  current  prior,  i.e.,  {s{n),S{n)).  The 
example  demonstrates  that  stocking  levels  will  generally  be  higher  when  only  sales  are 
observed  because  inventory  has  inherent  informational  value.  Thus,  the  example  illus¬ 
trates  the  concept  of  probing,  that  is,  stocking  more  inventory  to  get  a  more  precise 
estimate  of  demand  when  sales,  not  demand,  are  observed.  He  also  shows  when  man¬ 
agers  will  prefer  not  to  order  (policy  is  (0,0))  unless  more  information  is  received. 

Lovejoy  then  presents  a  finite  grid  approximation  method  to  tighten  the  bounds 
and  determine  suboptimal  policies.  First,  for  an  invariant  collection  of  sets,  the  function 
Ji{x,Tr)  is  piecewise  linear  and  convex  over  H.  This  property,  as  Sondik  and  Smaill- 
wood  [35]  discuss,  allows  new  vectors  to  be  added  to  the  set  in  order  to  improve  the 
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lower  bound.  Lovejoy  suggests  using  any  finite  set  of  points  in  11.  If  the  new  value 
functions  are  denoted  by  J'  and  HJ',  then  Ji  <  J'  <  H J'  <  J* .  The  discrete  points 
chosen  in  11  should  be  points  where  there  is  a  large  difference  between  the  upper  and 
lower  bounds.  At  each  dynamic  programming  rectusion  a  suboptimal  policy  is  generated 
to  improve  the  lower  bound,  and  the  upper  bounds  can  be  tightened  by  choosing  points 
that  form  a  triangulation  of  11  and  computing  jff„J(x,7r)  =  where  Vi 

represents  the  selected  points  and  Aj  are  such  that  tt  =  ^AjUj.  Then,  the  following 
results  hold:  H[HuJu]  <  HJu  <  Ju  and  if  J"  =  HuJu,  then  J*  <  HJ"  <  J". 

Lovejoy  extends  these  results  to  cases  in  which  some  parameters  are  nonstationary. 
The  underlying  demand  distribution  remains  stationary  and  the  horizon  length  is  finite. 
Again  he  shows  how  to  calculate  suboptimal  policies  and  then  iteratively  adjust  both 
the  upper  and  lower  bounds.  The  procedures  are  very  similar  to  those  for  the  stationary 
case,  except  that  the  procedure  must  account  for  changes  in  parameter  values  over  time. 

2.5  Conclusion 

Managers  can  maximize  profits  by  very  judicious  control  of  their  inventory.  To  do 
so,  they  to  not  need  to  automatically  assume  a  system  of  certainty  equivalence  control. 
This  task  has  become  increasingly  difficult  in  many  of  today’s  volatile  markets  which  are 
marked  by  high  levels  of  uncertainty.  Despite  the  inherent  difficulties,  there  are  practical 
tools  and  theory  available  to  eliminate  some  of  the  costly  effects  of  uncertainty.  This 
chapter  has  reviewed  several  optimal  control  policies  for  both  uncensored  and  censored 
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demand  observations  and  has  presented  a  summary  of  key  properties  for  the  solutions 
to  these  problems.  For  some  problems,  state  space  reductions  make  partial  information 
problems  as  solvable  as  their  perfect  information  counterparts.  In  most  practical  cases, 
however,  problems  must  be  solved  suboptimally. 

An  analysis  of  the  current  research  makes  it  evident  that  there  is  room  for  much 
more  research  in  this  area.  Perhaps  the  largest  area  of  potential  research  is  the  exten¬ 
sion  of  the  current  theory  and  models  to  “nonstationary”  demand  problems  in  which 
the  underlying  demand  distribution  changes  over  time.  Many  actual  inventory  control 
problems  are  characterized  by  partial  information  plus  nonstationary  demand.  This  ex¬ 
tension,  combined  with  partial  information,  complicates  the  calculation  of  an  optimal 
or  suboptimal  solution.  However,  with  the  advances  in  computer  memory  and  speed, 
solutions  to  these  problems  should  be  possible  with  well  designed  algorithms. 

Although  there  are  potentially  significant  financial  benefits  to  adaptive  control  pro¬ 
cedures,  several  fundamental  obstacles  prevent  wide  use  of  these  techniques.  First,  man¬ 
agers  lack  a  set  of  practical  tools  that  they  can  use  to  quantify  the  uncertainty  and  to 
adaptively  control  inventory  in  the  face  of  this  uncertainty.  Managers  must  be  provided 
these  tools  as  well  as  an  understanding  of  the  effect  of  the  assumptions  that  they  must 
make.  For  example,  managers  must  know  how  to  determine  the  a  priori  beliefs  for  the 
unknown  parameters.  While  doing  so,  they  need  to  know  how  this  assumption  affects 
the  solution  they  obtain,  i.e.,  the  Bayesian  robustness  of  the  prior.  Second,  managers 
often  make  parameter  estimates  and  then  rely  on  some  form  of  certainty  equivalence 
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control  (CEC).  Managers  must  understand  the  expected  level  of  error  when  using  such  a 
procedure.  For  stationary  problems,  the  adaptive  control  procedures  will  converge  to  a 
CEC  solution.  Thus,  one  must  know  when  it  is  economically  prudent  to  switch  from  an 
adaptive  control  procedure  to  a  CEC.  Once  a  CEC  is  used,  a  valid  procedure  is  needed 
to  identify  when  use  of  CEC  is  no  longer  economically  sound,  such  as  in  the  tail  of  a 
product  life  cycle.  Effective  tools  axe  needed  to  determine  when  such  changes  in  strategy 
should  be  made.  Third,  managers  may  choose  to  forego  adaptive  procedures  due  to 
the  restrictions  on  the  prior  and  demand  distributions.  For  the  continuous  parameter 
cases,  computational  resources  normally  require  that  the  demand  and  prior  distributions 
come  from  conjugate  families.  Managers  should  know  when  their  demand  process  can 
be  accurately  approximated  by  these  families  and  the  cost  of  assuming  that  they  do. 

Finally,  in  some  cases  there  may  not  be  a  conjugate  family  that  adequately  approx¬ 
imates  the  reaJ  problem.  In  these  cases,  a  bounded  discretized  prior  can  be  developed 
for  any  desired  level  of  accuracy  along  with  a  set  of  bounded,  discretized  demand  dis¬ 
tributions.  Using  a  discrete  model  and  solution  strategy  similar  to  that  of  Lovejoy  [28], 
the  posterior  distribution  can  now  be  easily  calculated  as  well  as  a  suboptimal  control 
policy.  Thus,  this  discretization  approach  constructs  a  policy  using  suboptimal  control 
for  accurate  prior  and  demand  distributions  whereas  the  continuous  modeling  approach 
applies  optimal  control  to,  in  these  cases,  less  accurate  prior  and  demand  distributions. 
An  interesting  area  of  research  would  be  to  determine  when  the  discretization  approach 
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provides  better  control  policies  than  the  continuous  model  when  the  necessary  distribu¬ 
tion  assmnptions  are  violated. 

We  have  reviewed  the  key  literature  for  this  adaptive  inventory  control  problem  and 
identified  some  of  its  salient  characteristics  and  areas  for  further  research.  The  problem 
area  is  important  for  a  large  number  of  actual  inventory  management  environments,  and 
this  number  continues  to  grow.  In  conclusion,  it  is  clear  that  there  are  many  interesting 
and  relevant  areas  of  research  for  this  fundamental  inventory  problem,  and  we  hope  that 


this  review  will  stimulate  that  research. 
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2. A  Optimality  of  (s,  S)  policy 


The  inventory  problem  with  fixed  purchase  costs,  linear  holding  and  penalty  costs, 
and  perfect  information  has  an  optimal  (s,  S)  policy.  Although  Kaplan  [16]  uses  an  (s,  S) 
policy  for  his  problem,  he  does  not  show  its  optimality.  Lovejoy  [28]  suggests  that  the 
proof  of  the  optimality  of  the  (s,  S)  policy  for  the  partial  information  problem  is  an 
extension  of  the  perfect  information  proof.  Bertsekas  [5]  provides  a  proof  for  the  perfect 
information  problem,  and  we  point  out  the  essential  parts  of  the  extension  of  this  proof 
to  the  partial  information  problem.  As  shown  in  Figmre  (2.3),  the  cost  function  is  not 
convex,  so  the  proof  strategy  is  to  show  that  for  all  t  and  tt^,  irt)  is  K-convex  in 

xt-  If  a  continuous  function  g  is  K-convex  and  g{y)  ->  oo  as  |y|  oo  then  by  Lemma 
2.1  in  Bertsekas  (page  149),  an  (s.  S')  policy  is  optimal.  That  is,  K-convexity  proves  the 
uniqueness  of  both  s  and  S.  Otherwise,  the  situation  in  Figure  (2.4)  could  occur  which 
would  imply  multiple  intervals  for  reordering. 

We  present  the  following  definition  and  lemma  from  Bertsekas  for  clarity. 

Definition.  A  real  function  g  is  K-convex  i/V  2;  >  0, 6  >  0,  K  >0,y,  then 
K-\-g{z  +  y)>  g{y)  + 


Lemma  2.1. 

(a).  A  real-valued  convex  function  g  is  also  0-convex  and  K-convex  for  all  iC  >  0. 

(h).  If  gi{y)  and  g2{y)  o.Te  K-convex  and  L-convex,  then  cxgiiy)  +  Pg2{y)  is  aK  +  /?L  - 
convex  for  all  a,/3  >  0. 

(c) .  If  g(y)  is  K-convex  and  w  is  a  random  variable,  then  E>u}{g{y'-w)}  is  also  K-convex 

if  Ew{\g{y  -  vj)\}  <  oo  for  all  y. 

(d) .  If  g  is  a  continuous  K-convex  function  and  g{y)  oo  as  \y\  — >  oo,  then  there  exist 

scalars  s  and  S  with  s  <  S  such  that 


Figure  2.3:  Cost  Function  with  Fixed  Order  Cost 


(1)  9{S)  <  9{y),  for  all  y; 

(2)  9{S)  +  K  =  9{s)  <  G{y),  for  all  y  <  S; 

(3)  g{y)  is  decreasing  on  (—00,5); 

(4)  9{y)  ^  9{^)  +  K  for  all  y,  with  s  <y  <  z. 


Let 


Gt{y\  n)  =  cy  +  H{y,  ttj)  +  Jt+i(y  -  w\  7r(+i)} 
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Figure  2.4:  Possible  Cost  Function  with  Fixed  Order  Cost 


where  H{y;Trt)  is  the  expected  holding  and  penalty  costs  in  period  t.  If  JT{xt’,T^T)  =  0, 
then  for  fixed  tt,  Gt-i  is  clearly  convex.  The  cost  function  is 

K  +  ^T-i)  —  cx  if  a:  <  st’-i; 

=  \  (2.21) 

GT_i(x;7rx_i)  —  cx  ifx>sr_i. 

Given  that  Gt-i  is  K-convex,  to  show  that  Equation  (2.21)  is  K-convex,  we  must  show 
that  V  2:  >  0, 6  >  0,  if  >  0,  y; 


K  +  Jr-iiy  +  z;TrT-i) 


— ivyj 


(2.22) 
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There  axe  three  cases  which  must  be  examined.  The  first  case  is  for  y  >  st-i-  If 
y  —  b  >  st-\,  then  Jr_i(xr-i;  is  clearly  K-convex  since  it  is  the  sum  of  a  K-convex 
function  and  a  linear  function  by  Equation  (2.21)  and  Lemma  2.1(b).  If  y  —  6  <  st-i, 
Equation  (2.22)  can  be  written  as 


K  +  GT-\{y  +  Z\’KT-l)  >  GT-\{y\T^T-l)  +z 


)■ 


(2.23) 


Then  y  is  such  that  GT-\{y\T^T-i)  is  either  >  or  <  G7’_i(sr-i; ’’’r-i)-  In  either  case, 
Equations  (2.22)  and  (2.23)  hold,  and  is  K-convex.  The  second  case  is 

for  y  <y  +  z  <  sr-i-  ttt-i)  is  linear  here  by  Equation  (2.21)  and  therefore 

K-convex.  The  third  case  is  y  <  st-i  <  y  +  z.  Again,  Equation  (2.22)  holds  and 
Jx_i(ir-i;  Tfr-i)  is  K-convex.  We  have  shown  that  due  to  the  K-convexity  of  Gt-i 
that  is  also  K-convex  as  well  as  continuous.  Extending  Lemma  2.1(c), 

if  y(y;  tt)  is  K-convex  for  fixed  tt  and  it;  is  a  random  variable,  then  Eyjigiy  —  w);  T(7r,  m)} 
is  also  K-convex  when  E.u}{\g{y  —  w)\;  T{it,  lo)}  <  oo.  Assuming  that  demand  is  bounded, 
then  the  expectation  over  tt  preserves  the  K-convexity.  Therefore,  Gt-2  is  also  K-convex, 
continuous,  and  GT-2iy',^T-2)  — )■  oo  as  |y|  — >■  oo.  We  can  show  that  for  all  t  that 
GtiViT^t)  is  K-convex,  continuous,  and  Gt{y,Ttt)  -t  oo  as  |y|  ->  oo  which  implies  that 
Jt{xt;nt)  is  also  K-convex.  By  Lemma  2.1(d),  an  (s,  S')  policy  is  optimal.  □ 


Chapter  3 


Policies  for  Non- Stationary  Demand  and  Partial  Information 

3.1  Introduction 

Product  demand  is  often  characterized  by  partially  observed,  non-stationary  de¬ 
mand.  The  demand  process  is  partially  observed  because  the  distribution  of  demand  in 
a  given  period  is  not  known  with  certainty,  and  it  is  nonstationaiy  because  the  distri¬ 
bution  may  randomly  change  over  time.  When  these  conditions  exist,  managers  must 
estimate  the  unknown  demand  distribution,  and  often  some  type  of  certainty  equivalent 
control  (CEC)  that  assumes  the  estimate  is  exact  is  used  to  solve  the  problem.  We 
present  compelling  research  that  demonstrates  that  there  exist  other  practical  subopti- 
mal  control  policies  to  solve  realistic  instances  of  this  problem  without  assuming  such  an 
estimate.  Furthermore,  these  control  policies  achieve  much  better  performance  in  some 
cases  than  the  CEC  policies  found  in  practice.  These  suboptimal  procedures  include 
Open-Loop  Feedback  Control  and  Limited  Look-Ahead  Control.  Both  of  these  policies 
account  for  more  of  the  inherent  uncertainty  in  the  demand  process  and,  therefore,  often 
outperform  the  CEC  policies.  We  examine  a  finite  horizon  problem  with  positive  lead 
time,  full  backordering,  and  linear  holding  and  backorder  costs.  We  model  the  demand 
process  as  a  composite-state,  partially  observed  Markov  decision  process.  The  demand 
distribution  is  assumed  to  belong  to  a  known,  discrete  set  of  candidate  distributions.  A 
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known,  discrete  a  priori  distribution  is  assumed  for  this  set  that  is  used  along  with  sub¬ 
sequent  observations  to  compute  a  posterior  distribution.  We  test  these  control  policies 
over  a  wide  range  of  problem  instances  and  compare  them  with  an  optimal  policy  and 
three  CEC  policies. 

3,2  Literature  Review 

The  inventory  control  literature  can  be  categorized  into  four  classes.  First,  there  are 
problems  with  complete  state  information  and  stationary  demand  which  form  the  body 
of  most  of  the  classical  inventory  theory.  Lee  and  Nahmias  [23]  summarize  this  classical 
theory  quite  well.  Secondly,  there  are  problems  with  complete  information  and  non¬ 
stationary  demand.  These  problems,  although  they  axe  more  complex,  have  been  studied 
fairly  well  also.  For  example,  Anupindi,  Morton,  and  Pentico  [2]  provide  four  heuristics 
for  a  nonstationary  stochastic  lead  time  problem.  Much  less  research  is  available  for 
the  other  two  categories  of  problems,  those  with  partial  information  and  stationary  or 
non-stationary  demand.  This  chapter  will  focus  on  the  case  of  non-stationary,  stochastic 
demand  with  incomplete  information.  But  first,  we  review  some  of  the  literature  for 
related  problems. 

One  of  the  earliest  cases  of  a  stochastic  non-stationary  demand  problem  is  found 
in  Hadley  and  Whitin  [13].  They  formulate  a  solution  to  a  final  inventory  problem  for 
which  there  is  a  known  obsolescence  date  and  Poisson  demand.  Shortly  prior  to  Hadley 
and  Whitin,  Karlin  [17]  analyzes  a  dynamic  system  (non-stationary  and  stochastic)  in 
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which  the  distribution  of  demand  can  vary  each  period,  and  it  is  an  extension  of  the 
Arrow-Harris-Marshak  [3]  dynamic  inventory  model.  Karlin’s  main  contribution  is  to 
qualitatively  show  how  the  critical  number  (optimal  inventory  level)  varies  over  time 
as  a  function  of  the  demand  densities.  He  determined  that  when  the  demand  densities 
decrease  stochastically  in  consecutive  periods,  the  critical  number  decreases.  If  the  den¬ 
sities  increase,  the  critical  number  may  or  may  not  increase.  Finally,  if  the  densities 
increase  in  successive  periods  while  the  critical  number  decreases,  then  the  critical  num¬ 
ber  decreases  further  in  the  following  period.  In  his  final  section,  he  discusses  cases  of 
“partial  information”  in  which  an  unknown  parameter  of  the  demand  distribution  has  an 
a  •priori  distribution  and  the  distribution  is  stationary.  He  analyzes  some  cases  in  which 
a  single  sufficient  statistic  exists,  namely  the  exponential  family  (including  gamma,  Pois¬ 
son,  and  negative  binomial)  and  the  range  family.  Kaxlin  then  presents  four  theorems 
in  which  he  shows  characteristics  of  optimal  policies  for  both  the  exponential  and  range 
families.  Veinott  [42]  extended  Karlin  [17]  by  showing  that  when  one  density  is  a  trans¬ 
late  of  another,  it  is  possible  to  relax  the  requirement  for  stochastic  ordering  to  show  the 
relationship  between  two  sets  of  critical  numbers.  He  also  shows  how  to  change  a  zero 
lead  time  problem  with  no  backlogging  to  one  with  backlogging. 

More  recently.  Song  and  Zipkin  [37]  discuss  a  general  modeling  framework  in  a 
fluctuating  demand  environment.  They  discuss  characteristics  of  optimal  policies  and 
describe  two  different  algorithms  to  solve  the  linear  cost  problem.  Their  paper  showed 
that  optimal  policies  depend  on  the  unit  cost,  holding  cost,  and  penalty  cost  as  in 
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the  newsvendor  problem.  Later,  Song  and  Zipkin  [38]  demonstrate  how  inventories 
should  be  managed  in  the  face  of  possible  obsolescence.  They  assume  that  a  continuous 
time,  non-increasing  discrete  state  Markov  chain  describes  the  demand  process.  One 
critical  assumption  they  make  is  that  the  current  demand  state  is  always  known  exactly. 
They  calculate  and  compare  an  optimal  policy  against  two  heuristics.  One  heuristic, 
a  blind  policy,  forecasts  demand  over  a  lead  time  assuming  that  there  is  no  change  in 
demand  over  the  lead  time.  The  second  heuristic,  a  limited  look-ahead  policy,  manages 
demand  over  a  lead  time  and  accounts  for  demand  changes  over  the  lead  time  but  not 
beyond.  Stephen  Graves  [12]  proposes  an  adaptive  base-stock  inventory  policy  for  a  non¬ 
stationary  problem.  His  demand  model  is  an  integrated  moving  average  model  of  order 
(0,1,1):  di=  n  +  e\  and  dt+i  =  dt  -  (1  -  a)et  -I-  et+i  for  t  =  1,  2,  . . .  .He  characterizes 
results  for  his  assumed  policy  without  asserting  its  optimality,  and  he  uses  an  exponential 
smoothing  procedure  to  estimate  the  demand.  This  problem  has  complete  information 
because  the  distribution  of  the  demand  in  a  period  is  fully  determined  by  the  ARIMA 
model  and  the  observed  demand  in  the  previous  period. 

Probleros  with  partial  information  are  significantly  more  difficult  to  solve.  Station¬ 
ary,  partial  information  problems  may  be  solved  by  Bayesian  methods  or  other  adaptive 
procedmres.  Herbert  Scarf  [33]  was  a  pioneer  in  the  use  of  Bayesian  techniques  for  inven¬ 
tory  control.  He  described  an  inventory  problem  with  linear  holding  and  penalty  costs. 
The  demand  density  was  described  by  a  density  (^(^,  w)  with  unknown  parameter  w.  The 
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parameter  w,  however,  has  a  known  a  priori  distribution.  The  demand  density  was  re¬ 
stricted  to  the  exponential  family  because  a  single  sufficient  statistic,  S ,  can  summarize 
all  prior  demand  information.  For  the  exponential  family,  the  sufficient  statistic,  5,  is 
the  average  demand  over  the  previous  periods.  Scarf  shows  that  the  optimal  stock  level 
can  be  calculated  recursively  with  knowledge  of  the  current  inventory  level,  x,  and  the 
sufficient  statistic,  S.  In  a  subsequent  paper  [34],  he  shows  that  in  some  instances,  the 
critical  level  can  be  determined  as  a  function  of  one  variable  if  the  demand  distribution 
is  in  the  gamma  family.  Katy  Azoury  [4]  demonstrated  that  other  problems  can  also  be 
solved  with  a  one-dimensional  state  space  when  there  is  a  natural  conjugate  family.  If 
the  demand  distribution  has  a  fixed  dimension  sufficient  statistic,  and  the  distribution  of 
the  unknown  parameter  comes  from  a  conjugate  family,  then  the  posterior  distribution 
for  the  unknown  parameter  is  also  in  that  same  family.  She  then  presents  two  conditions 
that  show  the  model  can  be  reduced  to  a  one-dimensional  state  space  and  solved  by  a 
dynamic  program. 

Lovejoy  [25]  uses  a  myopic  parameter  adaptive  technique  and  a  simple  inventory 
policy  based  upon  a  critical  fractile.  Specifically,  he  uses  exponential  smoothing  and 
Bayesian  updating  of  parameter  estimates.  He  also  determines  bounds  on  the  value  loss 
relative  to  optimal  costs  when  using  his  policy.  Lovejoy  [28]  provides  additional  insights 
into  the  stationary,  partial  information  problem.  These  insights  deal  with  bounds  for 
certain  suboptimal  policies,  and  he  shows  how  it  is  possible  to  iteratively  tighten  these 
bounds.  He  also  addresses  a  problem  in  which  some  parameters  may  be  nonstationary. 
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although  the  underlying  demand  distribution  remains  stationary.  Lovejoy  introduces 
the  concept  of  a  composite-state  partially  observed  Markov  decision  process  in  which 
the  inventory  level  is  observed  completely  while  the  demand  distribution  is  observed 
partially. 

Gallego,  Ryan  and  Simchi-Levi  [10]  demonstrate  a  Min-Max  technique  for  analysis 
of  various  distribution  free  finite  horizon  models  for  which  the  distribution  is  specified 
by  a  limited  number  of  parameters  such  as  the  mean  and  variance  or  a  set  of  percentiles 
of  demand.  They  provide  a  polynomial  algorithm  to  minimize  the  maximum  expected 
costs  over  all  possible  demand  distributions  with  the  given  parameters. 

Lariviere  and  Porteus  [20]  consider  Bayesian  techniques  for  a  lost  sales  problem  in 
which  sales,  not  true  demand,  are  observed.  Because  of  lost  sales,  the  retailer  can  learn 
more  about  the  true  demand  process  by  holding  inventory  at  a  higher  level  initially  to 
quickly  establish  a  better  estimate  of  the  true  demand.  The  process  of  finding  conjugate 
distributions  is  more  difficult  when  there  is  censored  data  due  to  lost  sales.  Lariviere 
and  Porteus  show  that  a  Weibull  density  for  demand  with  a  gamma  prior  on  one  of  the 
unknown  parameters  provides  the  necessary  conditions  for  a  known  posterior  distribution 
on  demand.  Agrawal  and  Smith  [1]  also  examine  lost  sales  and  contend  that  the  negative 
binomial  family  provides  a  very  good  fit  for  actual  retailing  sales  data.  They  describe 
procedmes  to  estimate  the  unknown  parameter  of  the  negative  binomial  distribution. 
Nahmias  [30]  also  proposes  procedures  for  demand  estimation  in  the  case  of  lost  sales 
assuming  normally  distributed  demand.  Karmarkar  [18]  shows  that  inventory  problems 
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are  often  concerned  with  the  determination  of  a  fractile  of  the  demand  distribution  which 
is  equivalent  to  determining  a  service  level  and  demonstrates  one  way  to  estimate  the 
critical  fractile.  Pierskalla  [31]  studied  the  stochastic  demand  problem  in  which  demand 
is  stationary  but  with  an  unknown  termination  date.  He  shows  that  the  distribution  of 
the  date  of  obsolescence  can  greatly  affect  the  critical  number.  Finally,  Kaplcin  [16]  uses 
a  Bayesian  approach  to  update  demand  information  for  a  procurement  problem.  First, 
an  initial  procurement  is  made  before  a  new  item  is  put  into  service.  After  the  part 
enters  service,  demand  information  is  used  to  provide  a  more  accurate  forecast. 

Problems  with  non-stationary  demand  and  partial  information  present  an  even  more 
complex  situation.  Little  direct  work  has  been  done  in  this  area,  but,  there  is  great 
potential  impact  for  research  on  this  class  of  problems.  One  paper  that  considers  a 
problem  in  this  class  is  by  Kmrawarwala  and  Matsuo  [19].  They  present  a  growth  model 
to  estimate  the  parameters  of  a  demand  process  over  its  entire  life  cycle.  In  their  base 
case,  production  decisions  are  made  at  the  beginning  of  the  problem  for  the  entire  life 
cycle.  They  present  a  technique  to  initially  estimate  the  parameters  of  their  forecasting 
model.  However,  they  do  not  thoroughly  address  the  issue  of  revising  these  estimates 
using  new  observations. 

We  model  the  problem  of  non-stationary  demand  with  partial  information  as  a 
composite-state,  partially  observed  Markov  decision  process  (CPOMDP),  see  Section 
3.3.  Partially  observed  Markov  decision  processes  have  been  well  studied  in  the  past, 
but  with  few  applications  in  inventory  and  production  control.  However,  it  composes 
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a  body  of  theory  with  some  direct  application  to  our  problem.  Monahan  [29]  presents 
a  good  survey  of  POMDP.  Smallwood  and  Sondik  [35]  show  that  for  a  finite  horizon 
POMDP  maximization  problem  the  objective  function  is  piece- wise  linear  and  convex  in 
the  current  state  probabilities.  They  then  provide  a  dynamic  programming  algorithm 
that  uses  this  property  to  solve  the  problem.  Lovejoy  [24]  presents  monotonicity  results 
for  certain  POMDP  problems.  White  and  Sherer  [43]  present  three  algorithms  for  the 
POMDP  that  are  faster  than  that  of  Smallwood  and  Sondik.  However,  the  problem 
sizes  are  very  limited  in  their  study.  White  and  Sherer  [44]  propose  a  heuristic  in  which 
only  the  M  most  recent  observations  and  actions  are  used  to  make  the  current  decision. 
They  also  introduce  the  concept  of  ergodicity  and  discuss  how  this  factor  indicates  the 
potential  value  of  historical  information. 

3.3  Optimal  Control  Model  and  Insights 

We  first  state  our  basic  model  that  will  serve  as  a  reference  model  for  further  dis¬ 
cussion.  We  assume  that  the  demand  in  any  given  period  arises  fi"om  one  of  a  finite 
collection  of  N  probability  distributions.  However,  the  decision  maker  may  not  have 
complete  certainty  about  which  of  the  distributions  generates  the  demand  in  a  given  pe¬ 
riod.  Furthermore,  the  distribution  may  randomly  change  from  one  period  to  the  next. 
Any  pattern  of  transitions  firom  one  distribution  to  another  can  be  modeled.  For  this 
basic  model,  we  assume  that  the  planning  horizon  is  finite,  stockouts  are  always  backo¬ 
rdered,  the  order  leadtime  is  either  zero  or  a  known  positive  constant,  and  there  is  no 
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fixed  order  cost.  However,  this  model  can  be  extended  to  allow  a  positive  fixed  order  cost 
and  observed  or  unobserved  lost  sales.  If  lost  sales  are  observed,  the  stockout  penalty 
cost  will  normally  be  adjusted  accordingly.  If  the  lost  sales  are  unobserved,  the  revision 
of  the  prior  for  the  unknown  parameter  is  more  complicated.  Lost  sales  cases  also  require 
an  augmented  state  space  which  can  significantly  complicate  the  computation  of  costs. 
We  define  in  Table  3.1  the  notation  that  will  be  used  to  describe  the  model. 


N 

number  of  distributions 

M 

= 

maximum  demand 

S 

stock  level 

h 

= 

linear  holding  cost, 

P 

= 

linear  stockout  penalty  cost, 

c 

linear  procurement  cost. 

T 

= 

end  of  planning  horizon, 

1 

lead  time. 

dt 

= 

demand  distribution  in  effect  for  period  t. 

Pij 

= 

P[dt-\-i  =  j\dt  ~ 

P 

(pij),  the  transition  probability  matrix  for 

Wt 

= 

demand  realization  in  period  t. 

Tjk 

"a 

II 

II 

It 

= 

vector  of  information  available  up  to  period  t, 

'^it 

= 

II 

TTt 

= 

int), 

inventory  level  at  the  beginning  of  period  t. 

Ut 

= 

inventory  position  at  the  beginning  of  period  t, 

at 

= 

order  quantity  in  period  t. 

at,i 

= 

vector  of  outstanding  orders, 

Table  3.1:  Notation 


Although  we  specify  linear  holding  and  stockout  costs  in  this  basic  model,  many 
of  the  results  hold  for  convex  costs.  The  demand  state  process  {d*}  is  not  known  with 
certainty  and  is  referred  to  as  the  core  process.  It  is  modeled  a;S  a  finite  state  Markov 


57 


chain,  dt  €  {1,  -  •  ■  ,N},  governed  by  the  transition  matrix  P.  If  dt  =  j,  the  probability 
distribution  for  demand  in  t  is  rj,  i.e.,  P[wt  =  k\dt  —  j]  =  rjk  for  k  =  0,1,..., M. 
Although  we  assume  that  the  matrix  P  is  stationary,  the  model  could  be  readily  extended 
to  allow  a  non-stationary  transition  process.  Note  that  there  is  a  great  deal  of  flexibility 
in  modeling  the  core  process.  The  case  of  a  stationary  demand  process  is  modeled  using 
P  =  I,  the  identity  matrix,  while  an  upper  triangular  matrix,  P^,  or  a  lower  triangular 
matrix,  Pi,  can  be  used  to  model  a  demand  trend.  A  slight  modification  of  the  upper 
or  lower  triangular  matrices  can  be  used  to  model  cychcal  demand.  The  core  process 
is  partially  observed  through  the  measurement  process  {wt}.  However,  in  the  case  of 
unobservable  lost  sales  we  assume  that  the  core  process  is  observed  through  the  sales 
rather  than  the  actual  demand;  therefore,  the  measurement  process  is  difierent.  For  the 
basic  model,  the  vector  of  information  available  at  time  t\slt  =  (mo,  mi, ... ,  wt-i).  Also 
known  is  ttj  and  Xt  where  ttq  is  the  initial  prior  distribution  that  is  externally  specified. 
In  practice,  ttq  will  be  based  on  previous  experience  with  similar  products  or  preliminary 
market  analysis.  Rhenius  [32]  has  shown  that  the  prior  distribution  ixt  is  a  sufiicient 
statistic  for  It-  Therefore,  the  problem  can  be  viewed  as  a  Markov  decision  process  on 
the  state  space  [xt,  Trj).  The  vector  TTt  characterizes  the  current  belief  of  the  distribution 
of  dt  given  all  prior  observations  of  the  information  process  {mj}.  The  system  dynamics 
for  our  CPOMDP  model  are  described  in  Figure  3.1.  The  control  process  begins  with 
an  information  vector,  It,  which  includes  a  prior  distribution,  ttj,  for  period  t,  and  then 
selects  an  order  quantity,  at-  For  positive  lead  time,  an  order  placed  I  periods  in  the 
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Figure  3.1;  Control  Process 


past  is  received.  The  demand,  wt,  then  occurs,  and  the  inventory  costs  for  period  t  are 
incurred.  The  time  index  advances  to  t  +  1,  and  the  core  process  advances  from  dt  to 
dt-i-i  according  to  the  transition  matrix  P.  The  new  prior  distribution  -Kt+i  is  computed 
using  TTtiWt  and  P.  Note  that  our  model  differs  from  the  t5rpical  POMDP  formulation 
(Monahan  [29],  Bertsekas  [5])  in  that  the  most  recent  observation,  wt,  used  to  update 
TTt+i  is  an  observation  of  dt  and  not  an  observation  of  dt+i-  Therefore,  the  transition 
equation  for  our  model  is 


TTi.t+l  =  Ti{lTt\wt  =  k)  = 


En 

j=l  ‘^jt1~jkPji 


(3.1) 
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If  we  assume  that  all  stockouts  are  backordered  and  that  the  leadtime  is  zero,  the  tran¬ 
sition  equation  for  the  inventory  level  is 


xt+i  =xt  +  at-  Wf  (3.2) 

For  an  inventory  level  u,  define  the  single  period  expected  inventory  cost  function  as 

Gt{v\-Kt)  =  -  u}  -I-  /imax{0,u  -  ty*}];  (3.3) 

then  the  optimal  stochastic  dynamic  programming  cost-to-go  function  is 

Jt{xun)  =  +  Gt{xt  +  at\'Kt)  +  E^^\^^[Jt+i{xt  +  at  -  wt,T{7rt\wt))]}, 

t  =  0,...,T,  (3.4) 

where  t  =  0  indexes  the  first  period  and  the  sequence  of  calculations  is  T,  T  —  1, . . . ,  1, 0. 
The  terminal  cost  function  is  defined  as  JT+i{xT+i,'n'T+i)  =  —cxt+i  so  that  there  is 
no  lost  value  for  the  terminal  inventory.  However,  the  results  for  this  model  apply  if 
Jr+iixT+i^T^T+i)  is  a  convex  function  of  xt  and  is  a  piece- wise  hnear  concave  function 
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of  TTt.  Alternatively,  if  we  define  St  =  xt  +  at,  the  problem  can  be  stated  as 

Jt{xt,Trt)  =  -cxt  +  min  {cSt  +  Gt{St\irt)  +  -£^u)t|7rJ'^t+i(<S't  -  wt,T{'Kt\wt))]), 

St^xt 

t  =  0,...,T.  (3.5) 

Our  model  permits  a  positive  and  known,  integer  leadtime,  I  >  0.  We  assume  that 
the  delivery  quantities  are  equal  to  the  order  quantities  and  are  completely  known.  This 
extension  requires  that  we  augment  the  state  space  to  include  the  outstanding  orders  at 
time  t.  Define  the  vector  at,i  =  (ot-i^  oe-i+i>  •  •  •  then  the  state  vector  at  time  t  is 

{xt,Trt,at,i).  The  inventory  transition  equation  is  xt+i  =  xt  +  at-i  —  wt,  and  the  order 
transition  equation  is  ct+i,/  =  at,i  —  {ot-;}  +  {oj}.  For  an  inventory  level  v,  define  the 
single  period  expected  inventory  cost  function  as 


GtivlnJ)  =  E^^ii„^\pmax{0,wt,i  -  v}  +  /imax{0,u  -  wt,i}],  (3.6) 

where  wt,i  =  X)n=o  ^i+n-  This  cost  function  accounts  for  the  positive  leadtime  in  com¬ 
puting  the  expected  inventory  costs  for  the  inventory  level  at  time  t  +  1.  The  non¬ 
stationary  nature  of  the  core  process  complicates  the  calculation  of  the  conditional  dis¬ 
tribution  (mt,i|7rt).  The  matrix  P"  is  the  n-step  transition  probability  matrix,  and  its 
transpose  is  denoted  (P")'.  For  n  =  1, . . . ,  /,  the  conditional  distribution  of  7r(+„  given 
TTt  can  be  computed  as  {'Kt+rMt)  =  and  the  conditional  distribution  of  lot+n 


61 


given  TTt  is  {wt+nM  =  {T^t+nlnYR  where  R  =  {rjk).  Finally,  the  conditional  distribu¬ 
tion  is  calculated  for  the  convolution  cost-to-go  function 

becomes 


Jt{xt,  TTt,  au)  =  min{cat  -1-  Gt{xt  +  at-iln)  + 

at>0 

Ewt\ni[Jt+i{xt  +  at-i-wt,T{Ttt\wt),at+i,i)]},  t  =  0,...,T.  (3.7) 

This  problem  can  be  simplified  by  formulating  it  in  terms  of  the  inventory  position, 
which  is  the  net  inventory  level  (on-hand  less  backorders)  plus  inventory  on-order  at 
the  beginning  of  t,  ut  =  xt  +  ot_„.  The  inventory  position  follows  the  transition 

equation  ut+i  =  ut  +  —  wt,  and  if  we  redefine  St  =  ut  +  at  then  the  DP  recursion 
becomes 

Jt{ut,  TTt)  =  -cut  -1-  nun  {cSt  +  Gt{St\nt,  1)  d-  Ey,A^^[Jt+i{St  -  wt,  T(7rt|t<;t))]}, 

t  =  0,...,T.  (3.8) 

It  is  important  to  characterize  the  cost  functions,  Jt{ut,'Kt)  and  the  optimal  policy. 
We  first  prove  the  convexity  of  the  cost  function. 

Theorem  1.  Jt(uf,7rt)  is  a  convex  function  of  ut  for  all  ttj. 

Proof:  The  proof  will  proceed  inductively.  First,  note  that  Jt-\-i{ut+i,  ttt+i)  =  —cut+i 
is  a  convex  function  of  ut+i-  Next,  assume  that  Jt+i(ut+i,7rf+i)  is  convex  in  ut+i,  then 
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we  will  show  that  Jtiut^nt)  is  also  convex  in  Ut-  Note  that  Gt{yt\T^t,l)  is  convex  in  yt 
and  that  lim|y^l_^oo  Gft(yt|7rt,Z)  =  oo.  Also,  note  that  because  T^t+i)  is  convex 

in  ut+u  Eyj^\^^[Jt+iiyt  -  wt,T{7rt\wt))]  is  also  convex  in  yt.  Therefore, 

Ht{yt\n,l)  =  cyt  +  GtiytlTTtJ)  +  E^t\'!Tt['^t+i{yt  -  wt,T{Trt\wt))] 

is  convex  in  yt-  Define  Stii^t)  as  the  value  of  yt  that  minimizes  Ht{yt\TttJ)-  If  “tH  <  St{T^t)i 
then  we  have  that 

=c{St{7rt)  -  Ut)  +  Gt{St{Trt)\nJ)  +  E^^\.^^[Jt+i{St{nt)  -  wt,T{nt\wt))] 

which  is  a  non-increasing  linear  function  of  ut.  However,  if  ut  >  StiTtt)  then 

Jt{ut,'JTt)  =  Gt(«t|7rt,Z)  +  E.u)t\‘irt['^t+i{ut  —  u)t.,T{T:t\u}t))] 

which  is  non-decreasing  and  convex  in  ut-  Because  T^t)  is  non-increasing  and  convex 
for  Ut  <  St{T^t)  and  non-decreasing  and  convex  for  ut  >  Sti'Kt),  it  is  convex  for  all  ut-  □ 
This  result  demonstrates  that  a  state-dependent  baise  stock  policy  is  optimal  for 
this  problem.  That  is,  there  exists  a  value  S't(7rt)  that  minimizes  the  function  Ht{yt\T^f,i) 
such  that  the  optimal  policy  is 


at  = 


St{n)  -  Ut,  if  Ut  <  Stin) 
0,  if  Ut  >  StiiTt) 
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This  result  allows  us  to  restrict  the  search  for  an  optimal  control  policy  to  the  class  of 
base  stock  policies  and  greatly  reduces  the  computational  requirements.  This  result  also 
justifies  the  use  of  base  stock  control  in  suboptimal  control  policies.  Although  the  optimal 
policy  has  a  relatively  simple  structure,  it  is  computationally  difficult  to  compute  the 
policy  because  the  stock  level  is  state-dependent.  Therefore,  we  describe  several  possible 
suboptimal  control  policies  in  the  next  section  that  can  be  computed  much  more  quickly. 

3.4  Suboptimal  Control 

In  this  section  we  discuss  suboptimal  control  policies  for  this  problem.  In  describing 
these  models,  we  assume  a  positive  lead  time  and  define  the  cost-to-go  function  using 
inventory  position  as  in  Equation  (3.8).  Because  of  the  intense  computational  require¬ 
ments  for  computing  optimal  policies,  it  is  valuable  to  develop  and  study  suboptimal 
control  policies  that  can  be  computationally  feasible  for  practical  application.  There 
are  many  different  forms  of  suboptimal  control  and  their  effectiveness  depends  a  great 
deal  on  the  problem  structure  and  parameters.  This  section  describes  several  different 
suboptimal  control  strategies  for  the  basic  model  described  in  Section  3.3.  We  assume 
that  a  control  policy  will  always  use  feedback  on  the  current  inventory  state  variable, 
xt  or  Uti  to  determine  the  order  quantity,  aj.  However,  the  policy  may  or  may  not  use 
feedback  to  revise  the  prior  distribution,  -Kt-  Therefore,  in  the  following  discussion,  the 
terms  open  loop,  closed  loop,  and  feedback  are  used  to  indicate  whether  or  not  a  control 
policy  uses  an  updated  prior,  tt*,  to  select  the  control  action  in  period  t. 
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3.4.1  Certainty  Equivalence  Control 

In  this  approach,  some  function  of  the  information  vector,  7(,  is  used  to  generate 
an  estimate,  dj,  of  dt-  Assuming  that  dt  =  dt,a.  complete  information  problem  is  solved 
to  determine  the  initial  base  stock  level  In  practice,  this  process  is  repeated  at  the 
beginning  of  each  period  using  the  newly  updated  information  vector.  This  approach  is 
essentially  the  one  used  by  most  inventory  control  systems.  The  estimate  dt  is  computed 
using  some  type  of  forecasting  model,  then  depending  on  the  problem  either  a  stationary 
or  non-stationary  demand  model  is  used  to  compute  5o.  This  approach  ignores  the 
uncertainty  in  the  estimate  dt  and  does  not  account  for  the  use  of  feedback  about  dt+n 
in  future  periods  when  calculating  the  cmrrent  solution. 

In  each  of  the  three  CEC  polices,  we  use  a  maximum  likelihood  estimate  (MLE)  to 
generate  dt-  For  the  first  policy  (CEC  1),  the  estimates  are: 

dt  =  argmax{7rit}  (3.9) 

i 

dt+n  =  argmax{P^-^^^_^  },  n  =  1, . . .  ,  T  -  t.  (3.10) 

The  estimate  at  time  t  is  the  distribution  that  is  most  likely  based  on  the  current  prior. 
In  subsequent  periods,  s  >t,  the  estimate,  dj,  is  the  distribution  that  is  the  most  likely 
given  ds_i. 
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For  the  second  CEC  policy  (CEC  2),  the  estimates  are: 


dt 


dt+n 


argmax{7rit} 

i 


n  =  1,...  ,T  —  t. 


(3.11) 

(3.12) 


The  initial  estimate  is  the  same  as  in  CEC  1.  However,  subsequent  estimates  are  a 
function  of  both  the  initial  prior  and  the  n-step  transition  matrix.  The  cost-to-go  function 
used  to  compute  the  stock  level  for  a  period,  t,  given  the  estimators  dt,  dt+i,  ■  •  •  ,  dr,  and 
the  current  inventory  position,  ut,  is 


jf+ni‘^t+n)  =  -CUt+n  +  _  min  I cSt+n  +  Gt+niSt+n\dt+n,  1) 

^wt+n\dt+n  (^<+n  ~  ■a’t+n)]  n  =  0, 1, . . .  ,T  —  t.  (3.13) 

where  Gt+n  is  calculated  from  Equation(3.6)  using  dt+n  instead  of  irt+n-  Note  that 
Equation  (3.13)  is  the  recursion  for  the  nonstationary,  complete  information  problem 
where  dt  is  known. 

In  the  above  two  CEC  policies,  the  estimate  at  any  given  time  is  one  of  the  N 
discrete  distributions.  In  the  third  CEC  policy  (CEC  3),  the  estimate  at  time  t,  dt, 
is  the  same  as  above,  but  subsequent  distributions  are  conditional  distributions  based 
on  dt  and  the  n-step  transition  matrix,  P".  The  distribution  in  subsequent  periods  is 
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determined  by  the  following: 

N 

P[wt-{.n  =  fc|dt]  = 

j=l 

The  cost-to-go  function  for  CEC  3  is 


J^ni'^t+n)  —  d”  nun  icSt-\.n  +  (3t-|_n(5^t+nMt^  0 

ot+n>Wt+n  ^ 

^Wt+n\dt  ^  y^t+n)]  j* j  n  =  0, 1, .  . .  —  t,  (3.14) 

This  procedure  is  similar  to  the  Open-Loop  Feedback  Control  discussed  next,  except 
that  the  demand  distributions  are  conditional  on  the  maximum  likelihood  estimate, 
rather  than 

3-4.2  Open-Loop  Feedback  Control 

Open-loop  strategies  do  not  use  feedback  to  update  estimates  of  TTt  for  t  >  0.  A 
decision  is  made  at  time  zero  for  an  initial  order-up-to  level  under  the  assumption  that  no 
feedback  will  be  used  in  the  future  to  calculate  new  stock  levels.  However,  in  practice, 
an  open-loop  control  problem  may  be  solved  each  period  with  the  previous  demand 
observation  used  as  feedback  to  update  the  current  prior.  This  type  of  control  may  be 
referred  to  as  Open-Loop  Feedback  Control  (OLFC)  (see  Bertsekas  [5]).  The  transition 
equation  =  T{7rt^i\wt-i)  updates  the  prior  at  the  beginning  of  and  the  cost-to-go 


function 
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J^ni'^t+n)  —  ~CUt+n  +  „  Dlin  -j  cSt+n  +  Gt+niSt+n\‘^t^l) 

St-\-n^y>t+n  ^ 

+  ^Wt+n\irt  [*^4+71+1  (“^t+n  “  ■i^i+n)]  | j  n  =  0, 1, . .  .  ,T  —  t,  (3.15) 

provides  the  new  base  stock  level,  St{nt).  This  approach  tahes  advantage  of  both  open 
and  closed-loop  control.  The  equation  for  assumes  that  feedback  will  not  be  used  in 
future  periods  (open  loop)  and  thereby  greatly  reduces  the  computational  requirements. 
However,  because  ttj  is  updated  using  the  prior  observations  (closed  loop),  it  does  incor¬ 
porate  the  information  available  at  t  to  characterize  the  distribution  of  Wf.  Another  way 
of  describing  this  approach  is  that  Equation  (3.15)  is  implemented  on  a  “rolling-horizon” 
basis. 

3.4.3  Limited  Look-Ahead  Control 

This  class  of  suboptimal  control  policies  optimizes  the  dynamic  problem  for  only 
a  limited  amount  of  time  into  the  future,  say  L  periods.  The  appropriate  transition 
equation,  irt  =  T{Trt-i\wt-i)  updates  the  prior  at  the  beginning  of  t.  When  L  =  0,  we 
refer  to  the  policy  as  a  myopic  policy  cind  the  approximate  cost  equation  is  defined  as 

J/'°(ut,  TTt)  =  nnn  {c{St  -  ut)  +  Gt{St\iTu  1)} ,  (3.16) 
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that  is,  the  base  stock  level  is  chosen  to  minimize  the  costs  only  for  the  current  period,  t. 
For  L  =  1,  we  refer  to  the  policy  as  the  single  period  look-ahead  policy  and  the  equation 
is 


=  min  {c(5t  -  ut)  +  Gt{St\n,l)  +  [JwiSt  -  Wt,T{7i-t\wt))] }  . 

(3.17) 

For  1/  =  2,  we  refer  to  the  policy  as  the  two  period  look-ahead  policy  and  the  equation 
is 


jP(ut,7rt)  =  min  {c(5t  -  n«)  -I-  GtiStlnJ)  +  [Jj^i  {St  -  wuT{7rt\wt))]  }  . 

St^ut 

(3.18) 

It  is  possible  to  look  even  further  ahead  into  the  future;  however,  the  computational 
requirements  increase  exponentially  as  the  look-ahead  period  increases.  Note  that  to 
solve  a  1-period  look-ahead  problem,  the  second  period  solution  is  a  myopic  policy. 
Similarly,  in  a  2-period  look-ahead  policy,  the  second  period  solution  is  a  single  period 
policy.  The  efficacy  of  these  strategies  and  the  best  choice  of  L  will  depend  upon  the 
dynamics  of  the  core  process  as  determined  by  the  transition  matrix  P. 
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Dist. 

Initial 
Mean  Std. 

Dev. 

Truncated 

Mean 

1 

1.00 

1.01 

1.00 

2 

9.00 

3.01 

8.97 

3 

16.00 

4.01 

14.20 

Table  3.2:  Demand  Distributions 


3.5  Experiment  Design 

In  this  section,  we  compare  the  results  of  an  optimal  poHcy  to  the  seven  suboptimal 
policies  described  in  the  previous  section.  There  are  three  discrete  distributions  that 
form  the  set  of  possible  distributions.  Each  of  the  distributions  were  initially  generated 
using  negative  binomial  distributions  and  then  truncated  at  a  maximum  18  aad  rescaled. 
Table  3.2  shows  the  demand  parameters.  We  list  the  demand  distributions  in  order  of 
increasing  mean  demand.  The  second  and  third  columns  show  the  mean  and  standard 
deviation  before  truncation,  and  the  fourth  column  shows  the  mean  of  the  truncated 
distributions.  Figure  3.2  shows  the  three  truncated  distributions.  In  Figure  3.3  we  show 
the  conditional  probability  of  each  distribution  after  a  single  observation  assuming  a 
uniform  prior  at  time  0.  If  the  observation  is  between  0  —  3, 5  —  9,  or  15  —  18  then 
there  is  a  high  probability  (above  80%)  that  the  current  state  is  distribution  1,  2,  or  3 
respectively.  Similarly,  a  demand  of  11,  12  or  13  will  not  provide  as  much  information 
about  whether  the  distribution  is  2  or  3.  Therefore,  a  single  observation  may  or  may  not 
provide  a  strong  indication  of  the  ciurent  state. 
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Figure  3.2:  Discrete  Demand  Distributions 


In  our  example  we  model  backordering  and  unit  cost  =  0.  Table  3.3  lists  the  pa¬ 
rameters  for  our  experimental  design.  For  each  of  the  three  lead  times  (0,1,  and  2), 
our  experimental  design  included  84  different  runs.  For  all  of  the  following  results,  the 
planning  horizon  is  5  periods.  Any  longer  horizon  length  makes  the  computational  re¬ 
quirements  for  computing  an  optimal  policy  excessively  long,  especially  with  a  lead  time 
of  2.  Procurement  cost  is  set  at  0,  and  the  lead  time  ranges  from  0  to  2.  We  examine  4 
different  priors  for  the  distribution  in  effect  at  time  0.  In  the  first  case,  the  distribution 
is  most  likely  in  the  high  distribution,  3.  In  the  second  case,  the  distribution  is  most 
likely  the  low,  1.  The  third  prior  indicates  that  there  is  no  strong  information  about 
the  current  demand  distribution.  Finally,  the  last  prior  indicates  a  strong  belief  that  the 
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Periods 

= 

5 

Leadtime 

= 

0 

1 

2 

Critical  Ratio 

= 

0.5 

0.65 

Prior  1  [S{7r{t) 

=  .4] 

= 

0.1 

0.3 

Prior  2  [c5(7r(i) 

=  .4] 

= 

0.6 

0.3 

■n 

Prior  3  [(5(7r(i) 

=  .0] 

= 

0.33 

0.34 

0.33 

Prior  4  [(J(7r(t) 

=  .7] 

0.1 

0.8 

Tran  1 

0.8 

0.1 

0.1 

[erg(P)=.7] 

0.1 

0.8 

0.1 

0.1 

0.1 

0.8 

Tran  2 

0.2 

0.4 

0.4 

[erg(P)=.2] 

0.4 

0.2 

0.4 

0.4 

0.4 

0.2 

Tran  3 

0.333 

0.333 

0.333 

[erg(P)=  0] 

= 

0.333 

0.333 

0.333 

0.333 

0.333 

0.333 

Tran  4 

0.8 

0.1 

0.1 

[erg(P)=.9] 

0.0 

0.8 

0.2 

0.0 

0.0 

1.0 

Tran  5 

0.2 

HI 

[erg(P)=.6] 

= 

0.0 

0.0 

Tran  6 

1.0 

0.0 

[erg(P)=.9] 

0.2 

0.8 

0.0 

0.1 

0.1 

0.8 

Tran  7 

1.0 

0.0 

[erg(P)=.6] 

= 

0.8 

0.2 

0.0 

0.4 

0.4 

Table  3.3:  Experiment  Parameters 
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<>  Distribution  1  □  Distribution  2  •  Distribution  3 


Figure  3.3:  Conditional  State  Probability  after  Demand  Observation 


distribution  is  the  middle  one,  2.  We  also  show  for  each  prior  the  values  for 


S(7rt)  = 


N 


2{N  - 1) 


N 


2=1 


N 


which  ranges  from  0  to  1.  The  function  6  measures  the  dispersion  of  the  prior  distribution 
TTt  from  a  discrete  uniform  distribution  on  {1, ,  N).  As  5  approaches  1,  there  is  a  high 
confidence  that  a  specific  distribution  is  in  efiect.  Similarly,  eis  5  approaches  0,  there  is  a 
low  confidence  that  a  specific  distribution  is  in  efiect.  We  experiment  with  7  transition 
matrices.  The  first  one  represents  a  very  stable  environment  and  the  second  represents 
a  highly  unstable  environment.  For  the  third  transition  matrix,  the  transitions  all  have 
equal  probability.  Matrices  four  and  five  represent  a  movement  to  a  permanently  high 
level  of  demand,  and  the  last  2  matrices  represent  movements  to  a  permanently  low 
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level  of  demand.  We  examine  three  critical  values:  .50,  .65,  and  .80.  These  ratios  were 
determined  by  keeping  the  holding  cost  constant  and  adjusting  the  penalty  cost  such  that 
CR  =  A  value  close  to  one  indicates  a  high  relative  penalty  cost  or,  equivalently, 
a  high  service  level.  It  should  be  noted  that  in  our  problem,  there  is  no  penalty  for 
underestimating  the  demand  in  subsequent  periods.  The  reason  for  this  is  that  there  are 
no  capacity  constraints,  so  if  the  current  inventory  level  (position)  is  below  the  order-up- 
to  level,  the  order-up-to  level  can  always  be  achieved.  On  the  other  hand,  it  is  possible 
to  move  into  a  subsequent  period  with  too  much  inventory.  It  is  in  these  instances  that 
a  suboptimal  policy  is  penalized.  It  would  not  be  dfficult  to  modify  this  problem  to 
penalize  a  situation  when  a  large  amount  of  inventory  must  be  produced  at  once.  This 
could  be  done  by  adding  capacity  constraints  or  convex  purchase  costs.  Finally,  under 
the  CEC  policies,  when  there  is  a  tie  between  2  distributions  most  likely  to  occur,  we 
choose  distribution  2. 

To  compare  the  policies,  we  adopted  the  following  procedure.  We  first  compute  the 
optimal  policy  for  each  parameter  set.  We  then  compare  the  optimal  expected  cost  to  the 
actual  expected  cost  of  each  suboptimal  policy.  Let  a  be  a  suboptimal  policy  with  state 
dependent  stock  levels,  5f(7rt),  and  order  quantities,  af(ut,7rt)  =  max{0, (vrt)  —  Ut}. 
Then  the  actual  expected  cost  of  a  can  be  computed  using  the  recursion 


Jt  (wt>  n)=c-  max{0,  St  (ttj)  -ut}  +  Gt(max{5'f  (TTt),  ut}|7rt,  1) 

+  Ewt\'Kt  [‘^i+i(niax{5'f  (7rt),itt}  -  wt,T{TTt\wt))]  ,  t  =  0, . . .  ,r.  (3.19) 
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Thus,  {uo,  TTo)  is  the  expected  cost  of  policy  a  over  the  given  horizon.  The  calculation 
requirements  to  compute  the  expected  costs  are  almost  as  intensive  as  they  are  to  com¬ 
pute  the  optimal  solution.  However,  it  is  unnecessary  to  compute  the  expected  costs  in 
practice. 


Lead  time 

Optimal 

Myopic 

LA  -  1 

LA  -  2 

OLFC 

CEC  2 

0 

Im  24s 

<ls 

<ls 

Is 

<ls 

<ls 

1 

7m  54s 

<ls 

<ls 

5s 

<ls 

<ls 

2 

Ih  47m  07s 

<ls 

5s 

Im  25s 

2s 

2s 

Table  3.4:  User  Runtime 


The  run  times  on  a  SUN  workstation  to  compute  six  policies  for  one  set  of  represen¬ 
tative  parameters  are  shown  in  Table  3.4.  The  optimal  column  shows  the  time  to  solve 
one  5-period  problem  optimally.  The  next  five  columns  show  the  time  to  solve  the  prob¬ 
lem  once  at  time  period  zero  only,  multiplied  by  five  to  account  for  the  five  decisions  that 
are  made  over  the  entire  horizon.  These  columns  show  times  for  the  myopic,  1-period 
look-ahead,  2-period  look-ahead,  open-loop  feedback  control,  and  the  second  certainty 
equivalent  control  policies.  The  calculation  of  the  exact  expected  costs  required  sub¬ 
stantially  more  time,  e.g.,  2.10  hours  of  user  time  on  the  same  system  for  a  2-period 
look-ahead  problem  with  a  lead  time  of  2. 

3.6  Results  and  Analysis 

The  complete  results  are  presented  in  Tables  3.11  through  3.19  of  the  appendix.  For 
emphasis,  values  of  “0”  are  indicated  by  blank  cells.  The  first  three  columns  designate 
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the  parameters  of  the  problem  instance.  The  next  column  shows  the  optimal  expected 
cost  for  the  5-period  problem.  The  remaining  columns  show  the  percentage  by  which 
each  suboptimal  policy  exceeds  the  optimal  cost.  Other  tables  are  organized  likewise. 
The  most  significant  result  is  that  the  full  consideration  of  the  uncertainty  of  the  demand 
distribution,  even  over  a  limited  horizon,  leads  to  substantial  improvements  over  the  CEC 
policies.  Even  the  myopic  policy  performs  substantially  better  than  the  CEC  policies. 

Table  3.5  shows  the  percentage  of  cases  in  which  each  suboptimal  policy  achieves  the 
best  cost  among  these  suboptimal  policies.  In  some  cases,  two  or  more  policies  tied  for 
the  best  suboptimal  cost.  Table  3.6  shows  the  average  percentage  that  each  suboptimal 


2  LA 

1  LA 

OLFC 

Myopic 

CEC  1 

97.62% 

50.00% 

44.84% 

43.25% 

1.59% 

1.59% 

1.19% 

Table  3.5:  Percent  of  Cases  Achieving  Best  Suboptimal  Cost 


policy  exceeded  the  optimal  cost.  In  this  and  other  summary  tables,  the  percentages 
reported  are  the  average  deviation  of  each  particular  case  from  its  respective  optimal 
cost.  The  results  in  Tables  3.5  and  3.6  lead  us  to  observe  that  the  best  suboptimal 
controls  based  on  cost  are,  in  order  from  best  to  worst,  2  LA,  1  LA,  OLFC,  Myopic, 
CEC  3,  CEC  2,  and  CEC  1.  OLFC  and  the  myopic  controls  are  fairly  close  in  their 
performance.  The  look-ahead,  myopic,  and  OLFC  control  policies  perform  much  better 
than  the  CEC  policies. 

We  summaxize  the  results  for  the  three  lead  times  in  Table  3.7.  We  first  see  that 


the  optimal  cost  goes  up  as  the  lead  time  increases.  This  is  certainly  expected  because 
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2  LA 

1  LA 

OLFC 

Myopic 

CEC  3 

CEC  2 

CEC  1 

0.33% 

1.53% 

2.03% 

5.46% 

12.56% 

16.68% 

18.04% 

Table  3.6:  Average  Percent  Above  Optimal  Cost 


Lead  time 

Cost 

Myopic 

ILA 

CECl 

CEC2 

CEC3 

$252 

1.31 

0.25 

14.08 

1 

5.74 

1.61 

0.37 

17.16 

14.32 

2 

$456 

1.66 

0.38 

22.27 

21.64 

11.33 

Table  3.7:  Summary  of  Results  by  Lead  Time 


demand  uncertainty  increases  as  the  lead  time  increases.  Note  that  the  cost  appears  to 
be  a  concave  function  of  the  lead  time.  The  three  look-ahead  policies  (Myopic,  1  LA, 
2  LA)  perform  quite  well  in  many  instances.  Even  the  myopic  policy  averages  between 
5  and  6  percentage  points  above  the  optimal.  For  the  1-period  look-ahead  policy,  the 
average  percentage  above  optimal  is  about  one  quarter  of  that  of  the  myopic  policy. 
The  consideration  of  only  one  additional  period  provides  substantial  improvement  over 
a  fairly  good  myopic  policy.  The  2-period  look-ahead  policies  are  less  than  one  quarter 
of  the  percentages  for  the  1-period  look-ahead.  The  2-period  look-ahead  policies  are 
typically  at  least  30-50  times  less  costly  than  the  best  CEC  policy.  Examination  of 
the  detailed  results  in  the  appendix  shows  that,  as  expected,  the  1-period  look-ahead  is 
always  at  least  as  good  as  the  myopic  policy  and  the  2-period  look-ahead  policy  is  always 
as  good  as  the  other  two  look-ahead  policies. 

The  OLFC  average  performance  is  comparable  to  the  1-period  look-ahead  policy. 
The  OLFC  never  performs  worse  than  the  myopic  policy,  although  it  is  usually  only 
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marginally  better.  This  is  because  the  OLFC  does  not  anticipate  feedback  to  revise  the 
prior  distribution  in  a  given  solution,  but  it  does  consider,  unlike  the  myopic  policy, 
the  uncertainty  in  the  future  due  to  transitions.  Although  the  OLFC  outperformed  the 
myopic  policy,  it  never  outperformed  the  l-period  look-ahead  policy.  This  result  suggests 
the  advantage  of  anticipating  feedback  to  solve  the  problem.  However,  given  that  the 
OLFC  may  be  much  faster  than  the  look-ahead  policies,  it  may  still  be  very  useful 
in  practice.  Also,  a  composite  look-ahead  and  OLFC  policy  might  provide  excellent 
performance.  For  example,  a  l-period  look-ahead  policy  might  be  used  with  an  OLFC 
for  the  remainder  of  the  horizon. 

Most  importantly,  the  CEC  policies  rarely  outperform  the  look-ahead  or  OLFC 
policies.  All  of  the  average  CEC  percentages  were  at  least  11%  above  the  optimal  cost. 
CEC  3  generally  outperforms  the  other  two  CEC  policies,  sometimes  very  significantly. 
The  improvement  in  CEC  3  relative  to  CEC  1  and  CEC  2  increases  as  the  lead  time 
increases.  This  is  because  after  time  t  the  future  demand  distribution  is  a  composite 
distribution  based  on  the  original  prior  as  well  as  the  transition  matrix.  Because  these 
or  similar  policies  are  often  used  in  practice,  these  results  suggest  that  even  a  very 
simple  policy,  such  as  the  myopic  policy,  that  fully  considers  the  demand  uncertainty 
will  perform  significantly  better  than  a  CEC  policy. 

We  used  three  difierent  critical  ratios  (see  Table  3.8).  We  see  that  the  optimal 
costs  go  up  as  the  critical  ratio  increases.  This  is  as  expected,  because  we  modified  the 
critical  ratio  by  keeping  the  holding  cost  constant  and  increasing  the  penalty  cost.  This 
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C.  R. 

Cost 

Myopic 

ILA 

CEC2 

CEC3 

0.50 

$263 

5.79 

1.31 

0.24 

2.54 

IQQ 

14.02 

12.46 

0.65 

$351 

5.76 

1.51 

0.36 

2.02 

14.96 

13.18 

10.78 

0.80 

$463 

4.81 

1.76 

0.40 

1.52 

22.04 

22.83 

14.44 

Table  3.8:  Summary  of  Results  by  Critical  Ratio 


equates  to  the  practice  of  requiring  a  higher  service  level  as  the  penalty  for  non-service 
increases.  The  optimal  cost  appears  to  be  a  convex  function  of  the  critical  ratio.  Again, 
the  2-period  look-ahead  policy  performs  extremely  well.  The  CEC  policies  generally 
perform  poorly.  CEC  3  outperforms  CEC  1  and  CEC  2  much  more  at  higher  critical 
ratios.  However,  if  one  keeps  all  parameters  constant  and  changes  only  the  critical  ratio, 
the  performance  of  any  given  suboptimal  policy  relative  to  another  cannot  be  predicted. 
For  example,  in  Table  3.11  the  CEC  policies  improve  with  prior  1  and  transition  matrix 
1  as  the  critical  ratio  increases,  but  worsen  as  the  critical  ratio  increases  with  prior  2 
and  transition  matrix  1. 

The  results  for  the  seven  transition  matrices  axe  summarized  in  Table  3.9.  Each  of 


P 

erg(P) 

ILA 

2LA 

OLFC 

1 

0.7 

$411 

1.41 

.35 

.07 

1.36 

20.19 

20.19 

11.17 

B 

0.2 

■ilgEkB 

36.82 

B 

0.0 

4.49 

4.49 

4 

0.8 

$320 

1.48 

0.39 

0.09 

1.48 

24.76 

24.38 

13.84 

5 

0.7 

$242 

6.61 

4.53 

5.43 

6 

0.9 

$410 

12.95 

4.12 

1.05 

10.00 

25.36 

18.28 

17.28 

B 

0.6 

$237 

22.36 

5.82 

1.12 

1.35 

14.00 

8.06 

12.61 

Table  3.9:  Summary  of  Results  by  Transition  Matrix 
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the  non-CEC  policies  achieve  the  optimal  cost  (or  very  close  to  it)  for  transition  matrices 
2,  3,  and  5.  These  matrices  represent  situations  when  the  transitions  are  highly  random 
(P  =  2,3)  or  rapidly  moving  to  a  permanently  high  demand  state  {P  —  h).  The  myopic 
policy  does  very  poorly  when  the  transitions  axe  to  a  lower  demand  level  (P  =  6, 7) 
because  there  is  a  very  strong  likelihood  of  carrying  excess  inventory  forward.  The  CEC 
policies  perform  very  poorly  when  the  transition  process  is  highly  unstable  (P  =  2). 
This  observation  leads  us  to  consider  the  following  measure  of  ergodicity  for  a  transition 
matrix  P  (see  Hopp,  Bean,  and  Smith  [14]) 

erg(P)  =  max  V  J  \pis  -  Pjs\  ■ 

^2 

This  quantity  is  a  measure  of  how  quickly  the  Markov  process  converges  to  its  stationary 
distribution.  When  erg(P)  is  close  to  1,  convergence  is  slow,  and  when  it  is  close  to 
zero  the  rate  of  convergence  is  fast.  Table  3.9  also  reports  the  ergodicity  values  for  each 
transition  matrix.  We  see  that  the  open-loop  feedback  control  policies  appear  to  perform 
better  as  the  ergodicity  measure  decreases  (P  =  2, 3).  It  appears  also  to  do  very  well  for 
P  =  5,  a  fast  transition  to  a  high  demand,  but  this  is  due  to  the  fact  that  there  is  no 
penalty  for  going  into  a  future  period  under-stocked.  Lower  ergodicity  values  usually,  but 
not  necessarily,  indicate  better  performance  for  limited  look-ahead  policies  also.  When 
the  ergodicity  value  is  high,  knowledge  of  the  current  state  of  the  core  process  is  much 
more  valuable  in  predicting  subsequent  demands.  Therefore,  adaptive  control  policies 
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that  update  tt*  using  the  current  observation  have  greater  benefit  over  non-adaptive 
policies  for  problems  with  a  higher  value  of  erg(P).  For  example,  with  transition  matrix 
6  (er^(P)  =  .9),  the  myopic  policy  is  12.95%  but  improves  drastically  to  4.12%  and 
1.05%  with  the  other  two  look-ahead  policies.  Furthermore,  suboptimal  policies  will  not 
perform  as  well  relative  to  optimal  policies  for  higher  values  of  erg(P)  and  such  problems 
will  provide  more  distinction  in  performance  among  suboptimal  policies. 


Pri. 

S(7r(t)) 

Cost 

Myo. 

ILA 

2LA 

OLFC 

CECl 

CEC2 

CEC3 

1 

■KOI 

$371 

MEM 

1.18 

0.36 

14.85 

$344 

■gg 

1.42 

0.21 

gigg 

15.28 

8.06 

2.60 

0.58 

ggg 

14.88 

15.14 

4 

2.60 

0.91 

0.18 

gogi 

12.31 

11.28 

Table  3.10:  Summary  of  Results  by  the  Prior 


Table  3.10  summarizes  the  results  for  the  four  difierent  priors.  The  suboptimal  poli¬ 
cies  generally  perform  the  worst  when  the  S(wt)  is  low,  because  there  is  little  confidence 
in  the  belief  of  which  distribution  is  in  effect  when  S(irt)  is  low.  The  poorest  results  for 
the  look-ahead  policies  are  usually  obtained  when  <5(7rt)  =  0.  There  are  some  exceptions, 
for  example,  in  Table  3.13  the  myopic  policy  does  very  poorly  when  there  is  a  transition 
to  a  low  demeind  distribution  (P  =  6, 7)  even  if  there  is  a  strong  belief  that  the  demand 
is  low  (prior  2).  This  is  because  if  the  demand  is  actually  in  a  higher  range,  there  is  a 
significant  penalty  to  pay  for  under-stocking.  The  1-period  look-ahead  policy  accounts 
for  this  and  substantially  improves  the  results  as  it  does  in  other  areas.  The  worst  case 
for  the  policies  that  explicitly  consider  the  prior  distribution  is  prior  3  which  indicates 
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no  current  information  is  available.  Naturally,  when  information  is  available  these  poli¬ 
cies  will  improve.  However,  the  worst  case  prior  for  the  CEC  policies  is  prior  2,  which 
indicates  a  strong  belief  that  the  demand  is  low.  This  is  because  the  CEC  policies  ignore 
the  possibility  that  the  demand  distribution  may  not  be  low  at  time  0  and,  therefore, 
they  under-stock  and  incur  high  penalty  costs. 

3.7  Conclusion 

We  have  demonstrated  that  a  policy  that  uses  feedback  and  fully  accounts  for  the 
demand  uncertainty  in  this  problem  can  substantially  outperform  a  certainty  equivalent 
policy.  Often  times  in  practice,  managers  feel  compelled  to  adopt  CEC  policies,  although 
this  is  frequently  not  necessary  and  often  very  costly.  By  incorporating  more  of  the  actual 
uncertainty  of  the  demand  process  even  over  a  limited  horizon  into  a  suboptimal  policy 
can  provide  dramatic  improvements  in  the  solution  of  this  problem.  In  many  cases 
even  a  myopic  policy  will  provide  significantly  better  results.  Additionally,  Open-Loop 
Feedback  Control  policies  also  will  often  perform  exceptionally  well  when  implemented 
in  a  “rolling  horizon”  fashion.  Look-ahead  policies  should  also  be  tractable  for  many 
realistic  problems. 

The  most  effective  policies  consider  the  uncertainty  of  the  demand  distribution 
through  an  a  priori  distribution.  This  prior  can  be  estimated  through  many  differ¬ 
ent  means  and  some  future  research  can  focus  on  efliicient  ways  to  obtain  these  priors  as 
well  as  evaluate  the  sensitivity  of  the  solution  to  the  accuracy  of  the  prior. 
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3.  A  Results 


Prior 

P 

CR 

Cost 

Myopic 

1  LA 

CEC  1 

CEC  2 

CEC  3 

1 

1 

$217 

1.10 

0.62 

■003 

0.99 

13.14 

1 

mm 

$269 

0.53 

0.36 

■■■■ 

0.52 

7.81 

7.81 

7.81 

1 

mm 

$324 

mHIQ2Qi 

0.14 

0.14 

0.17 

6.37 

6.37 

6.37 

2 

■■ 

0.50 

$178 

IHIZSI 

0.11 

0.01 

3.83 

3,83 

3.83 

2 

■■ 

0.65 

$263 

IIHKSSi 

0,30 

0.01 

6.19 

6.19 

2 

1 

$384 

0.58 

msgi 

0.58 

18.23 

3 

■1 

$216 

7.98 

hesi 

0.29 

7.92 

■■QOI 

■■^3 

12.29 

3 

a 

$291 

2,31 

0.79 

0.02 

2.31 

3.54 

3.54 

3 

■■ 

0.80 

$371 

0.81 

0.02 

4.93 

4.93 

4.93 

4 

1 

0.50 

$190 

1.21 

0.87 

0,02 

■BSEI 

3.74 

3.74 

3.74 

4 

1 

0.65 

$248 

0.03 

0-19 

2.03 

2.03 

4 

Bl 

$321 

0.07 

0.06 

4.96 

4.96 

1 

Bi 

$245 

■IQS3 

18.33 

21.05 

1 

B 

$312 

^■031 

1 

B 

$381 

86.10 

3II3QI 

2 

B 

■IIWIB 

$244 

20.93 

■■Q^l 

3II^£3 

2 

B 

0.65 

$323 

39.35 

■II3B 

■I^E3 

2 

B 

■Bl 

86.25 

86.25 

3 

B 

■IMIB 

■iitaara 

19.35 

■333 

3 

2 

0.65 

$328 

39.19 

38.56 

■333 

3 

B 

$402 

85.72 

86.72 

85.72 

4 

B 

$241 

20.93 

19.37 

20.93 

4 

B 

$308 

44.56 

44.04 

44.56 

4 

B 

$381 

96.45 

97.28 

96.45 

1 

B 

0.50 

$249 

3.19 

3.19 

3.19 

1 

B 

0.65 

$314 

3.36 

3.36 

1 

B 

$382 

13.97 

13.97 

^■££3 

2 

B 

$249 

0.47 

0.47 

2 

B 

$329 

5.17 

5.17 

2 

B 

$410 

26.46 

26.46 

26.46 

3 

B 

K£9i 

$261 

3 

B 

$332 

1.97 

1.97 

1.97 

3 

B 

$405 

16.18 

16.18 

16.18 

4 

B 

$241 

4 

B 

0.65 

$308 

1.70 

1.70 

1.70 

4 

B 

$381 

14.29 

14.29 

14.29 

Table  3.11:  Results:  Lead  time=0;  Trans  Matrices  1,  2  &  3 


83 


Prior 

rn 

CR 

Cost 

Myopic 

1  LA 

2  LA 

OLFC 

CEC  3 

1 

■■ 

■iWiM 

$157 

0.61 

11.44 

11.44 

11.44 

1 

Ei 

0.65 

$198 

0.30 

0.30 

^EESI 

7.60 

7.60 

1 

1.2iU 

0.16 

0.16 

iimQi 

0.16 

7.75 

7.75 

2 

4 

0.15 

0.01 

0.15 

3.06 

2 

4 

$234 

3.64 

0.29 

0.01 

3.64 

7.43 

2 

4 

0.63 

3 

4 

0.50 

9.26 

1.44 

0.33 

IKES 

11.73 

3 

■■ 

$242 

2.49 

0.84 

0.01 

2.49 

3.47 

3.47 

3.47 

3 

0.89 

6.97 

6.97 

6.97 

4 

■iiiifia 

0.73 

0.69 

0.73 

4.39 

4.39 

4.39 

4 

mm 

|i^ 

$206 

0.02 

0.02 

5.34 

5.34 

4 

wm 

liBilil 

$268 

13.31 

1 

a 

mMm 

6.35 

1 

Bi 

3.68 

3.68 

3.68 

1  1 

B 

l!^ 

1.78 

1.78 

2  ' 

B 

liBslil 

9.03 

9.03 

2 

B 

■iKiM 

$203 

2 

B 

$234 

KE9Q9 

keob 

3 

B 

lisa 

$155 

4.68 

4.68 

4.68 

3 

B 

l!^ 

$195 

3.35 

3.35 

3.35 

3 

B 

$236 

6.82 

6.82 

4 

B 

kb 

$131 

2.75 

2.75 

4 

B 

kb 

$166 

1.57 

1.57 

4 

B 

kb 

$205 

1.50 

1.50 

1.50 

Table  3.12:  Results:  Lead  time=0;  Trans  Matrices  4  &  5 
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Prior 

p 

CR 

Cost 

Myopic 

1  LA 

2  LA 

OLFC 

CEC  1 

CEC  2 

CEC  3 

1 

Ei 

mtmm 

$229 

4.99 

3.09 

0.94 

20.65 

1 

6 

0.65 

2.54 

0.27 

1 

■a 

$348 

1,35 

0.01 

1.31 

5.85 

5.85 

2 

6 

$123 

3.80 

1,61 

0.21 

3.57 

8.54 

8.53 

8,54 

2 

Ei 

K2&9 

1.31 

0.10 

6.01 

6.47 

6.47 

6.47 

2 

6 

■tMIM 

7.81 

0.60 

6.61 

6.61 

6.61 

3 

6 

giBsIsiH 

$192 

27.81 

4.87 

0.24 

3 

6 

0.65 

16.75 

7.72 

2,01 

16,65 

15.47 

11.70 

3 

llsiyjll 

8.51 

5.12 

2.45 

8.49 

2.91 

2.02 

2.91 

4 

6 

0.50 

ffRM 

7.57 

0.75 

7.45 

14.83 

14.64 

4 

Oi 

$246 

3.01 

mESi 

0.82 

2.61 

6.52 

6.39 

4 

Ei 

$319 

0.69 

2.18 

1.94 

1.85 

1.94 

1 

7 

0.50 

$147 

1,50 

1.69 

Hi99i 

16.50 

24.86 

1 

7 

0,65 

$218 

19.54 

0.67 

1.10 

1 

$322 

15.28 

4.28 

1.30 

1.58 

16.58 

13.47 

16.58 

2 

wm 

■IIW« 

$84 

0.57 

0.16 

0.16 

2 

7 

MiWsm 

$136 

0.95 

5.05 

5.05 

5.05 

2 

B 

■uiaiM 

$232 

0.81 

0.94 

8.46 

8,46 

8.46 

3 

B 

MKill 

$128 

44.35 

1.81 

1.82 

28.99 

28.99 

3 

B 

mmm 

$206 

31.14 

11.72 

2.80 

2.99 

13.26 

13.26 

3 

7 

$328 

21.27 

10.09 

2.11 

2.39 

6.43 

7.85 

4 

B 

9.57 

2.74 

2.74 

2.74 

4 

7 

0.65 

$148 

6.78 

4,53 

0.58 

0.65 

1.84 

1.84 

4 

7 

$226 

14.34 

4.85 

0.70 

0.83 

2.01 

2.01 

Table  3.13:  Results:  Lead  time=0;  Trans  Matrices  6  &  7 


Table  3.14:  Results:  Lead  time=l;  TVans  Matrices  1,  2  &:  3 
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Prior 

■a 

Myopic 

1  LA 

CEC  1 

CEC  2 

CEC  3 

1 

mm 

■iliM 

$227 

1.26 

0.38 

0.38 

1.26 

18.01 

17.20 

17.31 

1 

4 

0.65 

$288 

1.11 

0.25 

0,25 

1.11 

KKISI 

HBB^i 

1 

4 

$361 

0.14 

0.14 

0.14 

KKS^i 

8.88 

2 

Ki 

$267 

1.35 

0.31 

1.35 

17,66 

16.68 

16.34 

2 

4 

$360 

0.39 

0.39 

46.63 

44.96 

35.05 

2 

KI 

$474 

0.53 

0.53 

100.24 

97.84 

60.49 

3 

4 

$260 

3.43 

0.52 

3.43 

11.03 

30.09 

3 

KI 

$343 

4.48 

1.03 

0.16 

4.48 

7.85 

21.33 

3 

WSM 

$449 

2.84 

1.40 

0.45 

2.84 

9.55 

13.72 

4.80 

4 

mm 

KB 

$228 

0.96 

0.13 

10.82 

8.73 

7.86 

4 

4 

KB 

$297 

0.46 

0.46 

17.58 

13.91 

7.21 

4 

KI 

KB 

$385 

0.42 

0.42 

26.42 

1 

KB 

0.06 

0.06 

5.83 

5.83 

5.83 

1 

5 

$228 

2.61 

2.61 

2.61 

1 

B 

KB 

$286 

2.20 

2.20 

2.20 

2 

B 

KB 

$205 

16.35 

7.70 

KKI^I 

2 

B 

kb 

19.99 

4,21 

2 

B 

kb 

■Enanai 

6.25 

12.47 

3 

B 

ISEU 

$200 

5.81 

5.81 

4.95 

3 

B 

kb 

$258 

2.46 

2.46 

2.20 

3 

B 

kb 

$325 

1.66 

1.66 

4 

B 

kb 

$178 

2.57 

4 

5 

0.65 

$229 

WtBSSM 

0.82 

4 

B 

0.80 

$290 

0.93 

0.93 

0.78 

Table  3.15:  Results:  Lead  time=l;  Trans  Matrices  4  &  5 


Prior 

P 

CR 

Cost 

Myopic 

1  LA 

2  LA 

OLFC 

CEC  3 

1 

Ki 

6.49 

2.52 

0.97 

4.82 

41.88 

1 

wm 

iSlSQI 

3.12 

1.30 

0.34 

2.88 

21.51 

15.76 

1 

MM 

1.69 

0.24 

1.43 

13.24 

9.32 

2 

Q 

$199 

30.79 

2.07 

0.26 

20.33 

21,41 

19.54 

16.96 

2 

O! 

$290 

■009 

23.39 

2 

6 

$431 

■nnE« 

20.20 

16.69 

3 

WbMm 

$320 

Hi^Si 

■OKI 

■OKI 

22.47 

59.21 

22.49 

49.56 

3 

$452 

15.51 

9.17 

3.13 

12.43 

8.01 

3 

6 

■iI:CT 

$616 

10.23 

3.97 

1.14 

10.53 

8.31 

9.23 

4 

6 

0.50 

$309 

7.86 

3.25 

0.76 

■■SOI 

4 

Ki 

$412 

4.73 

1.81 

0.41 

HESSi 

13.94 

9.48 

4 

wm 

|1£Q| 

$532 

1.98 

0.87 

0.50 

1.68 

7.06 

6.61 

6.71 

1 

B 

$201 

24.62 

5.28 

2.23 

2.61 

70.14 

6.63 

1 

B 

$299 

18.49 

6.58 

1.99 

2.56 

38-21 

6.41 

1 

B 

■maiM 

$448 

17.57 

6.15 

2.43 

1.56 

21.07 

12.97 

2 

B 

$111 

5.76 

0.06 

0.03 

1.83 

1.83 

1.83  1 

2 

7 

$174 

38.64 

1.72 

0.03 

1.47 

8.75 

8.75 

2 

B 

$285 

34.04 

13.19 

0.51 

0.75 

3 

B 

$169 

45.89 

8,96 

0.01 

1.58 

15.45 

15.45 

15.45  1 

3 

7 

$265 

36.50 

12.02 

3.81 

4.18 

8.03 

8.03 

3 

B 

Kl^l 

$416 

22.31 

2.08 

2.29 

9.06 

4 

B 

EEd 

$132 

10.72 

0.63 

0.67 

0.86 

4 

B 

Kl^l 

$196 

0.75 

0.84 

2.54 

4 

B 

$300 

1.46 

0.33 

6.53 

3.59  1 

Table  3.16:  Results:  Lead  time=l;  Trans  Matrices  6  &  7 
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Prior 

P 

CR 

Cost 

Myopic 

1  LA 

2  LA 

OLFC 

CEC  1 

CEC  2 

CEC  3 

1 

1 

0.50 

$432 

0.75 

0.19 

1.90 

50.12 

HbwI 

1 

1 

$554 

1.47 

0.50 

0.09 

0.88 

33.82 

1 

1 

$696 

0,83 

0.21 

0.13 

0.82 

21.35 

21.35 

2 

1 

0.64 

0.07 

0.02 

0.60 

2 

1 

0.65 

0-66 

0.06 

0.01 

0.64 

BHE^i 

2 

1 

$687 

0.48 

0.03 

124.75 

3 

1 

$421 

4.09 

1.29 

0.09 

4.03 

28,12 

28.12 

3 

Bi 

0.65 

2.68 

0.67 

0.15 

2.64 

16,08 

16.08 

3 

mm 

BIE9 

1.90 

0.34 

0.28 

1.89 

8,75 

8.75 

5.46 

4 

mm 

0.50 

$381 

1.41 

0.36 

0.03 

1.38 

12.61 

5.77 

4 

0.65 

$494 

0.71 

0.11 

0.68 

7,35 

7.35 

4.15 

4 

$635 

0.59 

0.05 

5.94 

5.94 

1 

00 

1 

B 

$412 

10.90 

7.84 

1 

B 

$544 

14.77 

28.56 

1 

B 

E£9i 

$713 

61.58 

IIIlQiQI 

2 

B 

$413 

21.57 

2 

B 

$546 

32.55 

2 

B 

0.80 

$716 

25.75 

68.46 

12.31 

3 

B 

ESI 

10.03 

7.57 

3 

B 

0.65 

MiHgM 

8.77 

3 

B 

$722 

73.16 

11.60 

4 

B 

mimm 

$412 

9.96 

19.92 

8.14 

4 

B 

■IIRM 

$544 

15.36 

9.54 

4 

B 

■waiM 

$713 

25.93 

1 

B 

mtmam 

$414 

0.03 

4.86 

4.86 

1 

B 

mtKiim 

$547 

0.91 

1 

B 

$716 

1.75 

1.75 

0.49 

2 

B 

$415 

2.79 

2.79 

1.25 

2 

B 

$549 

2.14 

2.14 

2.14 

2 

B 

BISsU 

$721 

6.32 

6.32 

2.06 

3 

B 

$419 

3.25 

3,25 

0.20 

3 

B 

mtwtM 

$553 

0.12 

3 

B 

$724 

2.25 

2.25 

4 

B 

$411 

3.13 

3.13 

0.16 

4 

B 

0.65 

$543 

0.10 

4 

B 

kSBSSI 

$711 

Table  3.17:  Results:  Lead  time=2;  Trans  Matrices  1,  2  &  3 
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Prior 

m 

CR 

Cost 

Myopic 

1  LA 

2  LA 

OLFC 

CEC  1 

CEC  2 

CEC  3 

1 

wm 

$282 

1.21 

0.47 

0.07 

1.21 

21.78 

19.52 

19.99 

1 

K1 

$359 

0.05 

0.49 

17.31 

13.92 

1 

4 

Kl^l 

$451 

0.05 

0.60 

9.76 

2 

Ei 

■UWiM 

$331 

0.62 

0.01 

0.62 

48.28 

2 

KB 

$443 

0.91 

2 

4 

$595 

0.39 

0.39 

175.56 

165.34 

29.81 

3 

4 

$324 

5.24 

2.08 

0.34 

5.24 

16.85 

3 

4 

$428 

3.36 

0.28 

3.36 

9.82 

3 

wm 

DSSi 

$557 

0.09 

2.72 

14.68 

19.25 

6.23 

4 

KB 

$284 

0.55 

0.52 

0.06 

0.55 

19.52 

7.82 

9.20 

4 

4 

0.65 

$369 

0.33 

bees 

0.33 

9.71 

10.27 

4 

KB 

BSSI 

$477 

0.47 

0.47 

16.96 

10.83 

1 

B 

KSsI 

$211 

4.61 

4.61 

4.61 

1 

B 

EiESi 

3-60 

1 

5 

$346 

2.31 

2.31 

2.31 

2  1 

B 

K^l 

$238 

11.81 

6.47 

11.81 

2 

B 

ISgsgi 

$309 

13.37 

4.95 

10.75 

2 

B 

El^3i 

$399 

4.78 

3 

B 

$231 

6.41 

6.41 

3 

B 

EI^I 

$300 

3.05 

3.05 

2.74 

3 

B 

1023!! 

$382 

1.80 

1.81 

4 

5 

0.50 

$211 

2.95 

1.60 

4 

B 

EB 

$273 

1.47 

1.47 

1.32 

4 

B 

liB 

$349 

1.25 

1.25 

WMSMm 

Table  3.18:  Results:  Lead  time=2;  Trans  Matrices  4  &  5 
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Prior 

■a 

CR 

Cost 

Myopic 

1  LA 

2  LA 

OLFC 

CEC  1 

CEC  2 

CEC  3 

1 

ra 

$517 

5.97 

2.45 

1.10 

4.32 

70.58 

56.78 

35.89 

1 

0.65 

3.85 

1.25 

0.40 

2.53 

47.76 

36.35 

1 

■a 

$830 

1.99 

0.85 

0.21 

1.74 

29.26 

21.19 

2 

B 

$259 

34.25 

2.57 

0.29 

25.41 

35.39 

25.46 

16.09 

2 

B 

0.65 

$375 

35.20 

7.68 

28.44 

23.66 

13.48 

2 

6 

■MWta 

21.69 

3 

B 

$425 

3.12 

25.22 

45.67 

3 

B 

$602 

16.37 

7.53 

3.07 

13.71 

46.59 

13.69 

26.39 

3 

6 

3.40 

■Qggi 

6.40 

20.93 

18.84 

11.25 

4 

B 

0.50 

$407 

7.79 

3.59 

■Eia 

39.63 

19.82 

4 

6 

0.65 

$538 

4.34 

1.45 

0.38 

3.18 

IKSIS 

24.19 

13.75 

4 

B 

$697 

2.35 

0.79 

0.35 

2.03 

19.31 

14.81 

7.84 

1 

7 

gijfjjji 

6.92 

5.30 

37.76 

1 

7 

0.65 

$327 

6.89 

1.28 

HEI^i 

3.70 

28.66 

1 

B 

mmm 

$489 

14.89 

7.42 

1.93 

1.11 

15.53 

14.97 

22.90 

2 

7 

$125 

9.79 

0.08 

0.07 

2.35 

2.35 

2.35 

2 

7 

$193 

30.24 

0.51 

0.12 

0.23 

3.40 

3.40 

2 

B 

■maiM 

$306 

35.26 

9.80 

1.42 

1.61 

18.72 

18.72 

3 

B 

■ililiM 

$185 

38.37 

6.67 

0.76 

3.09 

12.47 

12.47 

18.28 

3 

B 

0.65 

$287 

30.74 

13.63 

2.44 

2.77 

4.69 

4.69 

7.14 

3 

7 

|[i2ts2j^ 

$445 

mgggi 

1.38 

10.77 

9.79 

4 

B 

fcllWIM 

$146 

8.73 

0.28 

0.56 

0.56 

2.75 

4 

B 

0.65 

$215 

15.01 

2.19 

0.12 

0.20 

0.58 

0.58 

0.68 

4 

B 

ll!£m 

$324 

17.11 

6.84 

0.54 

0.67 

9.23 

9.23 

4.85 

Table  3.19:  Results:  Lead  time=2;  Trans  Matrices  6  &:  7 


Chapter  4 


Bounds  and  Approximations  for  Adaptive  Inventory  Control 

4.1  Introduction 

In  practice,  product  demand  is  tjrpically  highly  uncertain  and  subject  to  change, 
and  the  demand  process  is  only  partially  observed  because  the  probability  distribution 
of  demand  in  a  given  period  is  not  known  with  certainty.  Moreover,  this  demand  dis¬ 
tribution  may  change  over  time  according  to  some  random  process.  Because  optimal 
solutions  for  this  problem  are  available  only  for  small  problems,  managers  must  esti¬ 
mate  the  unknown  demand  distribution.  This  estimate  is  commonly  treated  as  if  it  were 
known  with  certainty,  and  then  some  type  of  certainty  equivalence  control  strategy  is 
used  to  generate  a  control  policy. 

Effective  approximations  to  the  optimal  cost  function  are  known  for  the  stationary 
version  of  this  problem,  and  we  extend  these  procedures  to  demonstrate  that  similar 
techniques  can  be  used  when  the  demand  distribution  is  non-stationary.  We  consider 
an  infinite  horizon  problem  with  positive  lead  time,  full  badcordering,  and  linear  holding 
and  backorder  costs.  We  model  the  problem  as  a  composite-state,  partially  observed 
Markov  decision  process.  The  demand  distribution  is  assumed  to  belong  to  a  known 
and  discrete  set  of  candidate  distributions,  and  a  known,  discrete  o  priori  distribution  is 
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assumed  for  this  set  which  is  also  used  along  with  subsequent  observations  to  compute 
a  posterior  distribution. 

Optimal  solutions  to  this  problem  are  typically  intractable  because  the  state  space 
for  the  prior  is  uncountably  infinite.  Therefore,  we  first  develop  a  grid  approximation  that 
is  asymptotically  exact  for  this  problem.  We  then  show  how  to  calculate  both  an  upper 
bound  and  a  lower  bound  for  the  optimal  cost  function,  and  we  compare  these  bounds 
to  the  functional  approximation.  For  a  stationary  problem,  the  lower  and  upper  bounds 
are  tight  at  the  extreme  points.  However,  for  the  non-stationary  problem,  there  is  a  gap 
between  the  upper  and  lower  bounds  at  the  extreme  points.  Furthermore,  the  initial 
bounds  axe  not  as  tight  away  firom  the  extreme  points  as  in  the  stationary  problem  due 
to  the  added  uncertainty  generated  by  the  non-stationary  aspect  of  the  demand  process. 
However,  both  of  these  bounds  can  be  iteratively  tightened.  Additionally,  we  describe 
details  for  myopic  and  look-ahead  policies  for  this  problem. 

In  Section  4.2  we  review  key  literature,  and  in  Section  4.3  we  review  the  basic  control 
model.  In  Section  4.4  we  discuss  both  the  grid  approximation  as  well  as  upper  and  lower 
bounds  and  the  iterative  tightening  of  those  bounds,  and  the  myopic  and  look-ahead 
policies.  In  Section  4.5  we  provide  an  example  of  our  technique,  and  finally,  we  provide 


conclusions  and  recommendations  for  future  research  in  Section  4.6. 
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4.2  Literature  Review 

Non-stationary,  partial  information  problems  are,  by  their  very  nature,  difficult  to 
solve.  Therefore,  the  literature  for  this  class  of  problems  is  relatively  sparse.  Most 
traditional  inventory  control  models  assume  either  stationary  demand  or  complete  in¬ 
formation  about  the  underlying  demand  state  or  both.  Hadley  and  Whitin  [13]  solve  a 
stochastic  non-stationary  problem.  They  determine  a  final  inventory  level  for  a  known 
obsolescence  date  with  Poisson  demand.  Karlin  [17]  studies  non-stationary  problems  and 
presents  several  qualitative  characteristics  of  the  optimal  inventory  level.  For  example, 
he  notes  that  the  critical  level  will  decrease  if  the  demand  densities  are  decreasing  over 
consecutive  periods  but  may  or  may  not  increase  if  the  demand  densities  are  increasing 
over  consecutive  periods.  In  this  same  work,  he  assumes  that  the  demand  is  stationary 
but  that  the  distribution  is  only  known  partially,  that  is,  the  distribution  has  an  un¬ 
known  parameter  with  a  known  a  priori  distribution.  He  presents  several  results  about 
the  characteristics  of  optimal  policies  for  the  exponential  (including  gamma,  Poisson, 
and  negative  binomial)  and  range  families,  both  of  which  have  a  single  sufficient  statistic 
for  the  unknown  parameter.  Veinott  [42]  extended  Karlin’s  work  by  showing  that  when 
one  density  is  a  translate  of  another,  it  is  possible  to  relax  the  requirement  for  stochastic 
ordering  to  show  the  relationship  between  two  sets  of  critical  numbers.  He  also  shows 
how  to  change  a  zero  lead  time  problem  with  no  backlogging  to  one  with  backlogging. 

More  recently.  Song  and  Zipkin  [37]  present  a  general  modeling  firamework  for  non- 
stationary  demand.  They  demonstrate  that  optimal  policies  behave  very  similarly  to 


94 


the  newsboy  problem  in  their  dependence  on  unit  cost,  holding  cost,  and  penalty  cost, 
and  they  present  two  algorithms  to  solve  the  linear  cost  problem.  In  a  later  work, 
Song  and  Zipkin  [38]  examine  a  problem  with  obsolescence.  But  they  always  assume 
that  the  current  demand  distribution  is  always  known  with  certainty.  They  model  the 
demand  process  as  a  continuous  time,  non-increasing  discrete  state  Markov  chain.  They 
demonstrate  the  performance  of  a  blind  policy  which  forecasts  demand  only  over  the 
lead  time  and  assumes  that  the  demand  distribution  does  not  change  over  the  lead  time. 
They  also  examine  a  limited  look-ahead  policy  that  accounts  for  demand  changes  over 
the  lead  time  but  not  beyond.  Stephen  Graves  [12]  presents  an  ARIMA  based  model  to 
compute  an  adaptive  base  stock  policy  for  non-stationary  demand.  Again,  this  problem 
can  be  assumed  to  have  complete  information  because  the  distribution  of  the  demand 
in  a  period  is  fully  determined  by  the  ARIMA  model  and  the  observed  demand  in  the 
previous  period.  Gavirneni  and  Tayur  [11]  present  an  efficient  procedure  to  compute  the 
state  dependent  policy  for  a  non-stationary  problem.  They  do  not  calculate  the  actual 
cost  of  the  policy  but  rather  compute  the  derivative  of  the  cost  function  to  determine 
the  optimal  base  stock  level  for  each  state.  However,  the  optimal  cost  can  be  calculated 
using  other  methods  once  the  optimal  policy  is  known. 

Partial  information  problems  are  more  difficult  to  solve  than  the  complete  infor¬ 
mation  problems  addressed  by  the  papers  above.  Bayesian  methods  or  other  adaptive 
procedures  can  be  used  to  solve  these  problems,  but  they  often  require  a  great  deal  more 
computational  effort.  Herbert  Scarf  [33]  was  a  pioneer  in  the  use  of  Bayesian  techniques 
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for  inventory  control  when  he  studied  an  inventory  problem  with  linear  holding  and 
penalty  costs.  The  demand  density  was  described  by  a  density  with  unknown 

parameter  uj.  The  parameter  w,  however,  has  a  known  a  priori  distribution.  The  demand 
density  was  restricted  to  the  exponential  family  because  a  single  sufficient  statistic,  S', 
can  summarize  all  prior  demand  information.  For  the  exponential  family,  the  sufficient 
statistic,  S,  is  the  average  demand  over  the  previous  periods.  Scarf  shows  that  the  op¬ 
timal  stock  level  can  be  calculated  recursively  with  knowledge  of  the  current  inventory 
level,  r,  and  the  sufficient  statistic,  S.  In  a  subsequent  paper  [34],  he  shows  that  in  some 
instances,  the  critical  level  can  be  determined  as  a  function  of  one  variable  if  the  demand 
distribution  is  in  the  gamma  family. 

Katy  Azoury  [4]  demonstrated  that  other  problems  can  also  be  solved  using  a  one¬ 
dimensional  state  space  when  there  is  a  natmral  conjugate  family.  If  the  demand  dis¬ 
tribution  has  a  fixed  dimension  sufficient  statistic,  and  the  distribution  of  the  unknown 
parameter  comes  from  a  conjugate  family,  then  the  posterior  distribution  for  the  un¬ 
known  parameter  is  also  in  that  same  family.  She  then  presents  two  conditions  that 
show  when  the  model  can  be  reduced  to  a  one-dimensional  state  space. 

Lovejoy  [25]  uses  a  myopic  parameter  adaptive  technique  and  a  simple  inventory 
policy  based  upon  a  critical  firactile.  Specifically,  he  uses  exponential  smoothing  and 
Bayesian  updating  of  parameter  estimates.  He  also  determines  bounds  on  the  value  loss 
relative  to  optimal  costs  when  using  his  policy.  Lovejoy  [28]  provides  critical  insights  into 
the  stationary,  partial  information  problem  that  address  bounds  for  certain  suboptimal 
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policies,  and  he  shows  how  it  is  possible  to  iteratively  tighten  these  bounds.  He  also 
addresses  a  problem  in  which  some  parameters  may  be  non-stationary,  although  the 
underlying  demand  distribution  remains  stationary.  Lovejoy  models  the  problem  as 
a  composite-state  partially  observed  Markov  decision  process  (POMDP)  in  which  the 
inventory  level  is  observed  completely  while  the  demand  distribution  is  observed  partially. 
This  work  extends  this  approach  to  the  problem  in  which  the  demand  distribution  is  non- 
stationary. 

Problems  with  non-stationary  demand  and  partial  information  present  an  even  more 
complex  situation.  Little  direct  work  has  been  done  in  this  area,  but  there  is  great 
potential  impact  for  research  on  this  class  of  problems.  One  paper  that  considers  a 
problem  in  this  class  is  by  Kurawarwala  and  Matsuo  [19].  They  present  a  growth  model 
to  estimate  the  parameters  of  a  demand  process  over  its  entire  life  cycle.  In  their  base 
case,  production  decisions  are  made  at  the  beginning  of  the  problem  for  the  entire  life 
cycle.  They  present  a  technique  to  initially  estimate  the  parameters  of  their  forecasting 
model.  However,  they  do  not  thoroughly  address  the  issue  of  revising  these  estimates 
using  new  observations. 

Treharne  and  Sox  [40]  examine  an  inventory  control  problem  in  which  the  demand 
process  is  partially  observed  and  non-stationary.  Although  managers  often  use  certainty 
equivalent  control  (CEC)  policies  to  solve  such  a  problem,  Treharne  and  Sox  demonstrate 
that  there  exist  other  efficient  and  often  superior  suboptimal  control  policies  to  solve 
this  problem.  They  model  the  problem  as  a  composite-state,  partially  observed  Markov 
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decision  process  over  a  finite  horizon.  They  show  that  consideration  of  the  uncertainty 
of  the  demand  distribution,  even  using  a  suboptimal  policy  over  a  finite  horizon  often 
leads  to  substantial  improvements  over  CEC  policies. 

We  model  this  inventory  control  problem  as  a  composite-state,  partially  observed 
Markov  decision  process  (CPOMDP),  see  Section  4.3.  In  a  CPOMDP,  some  states  are 
fully  observed  while  others  are  only  partially  observed.  Partially  observed  Markov  deci¬ 
sion  processes  have  been  well  studied  in  the  past,  but  with  few  applications  in  inventory 
and  production  control.  However,  it  presents  a  body  of  theory  with  some  direct  applica¬ 
tion  to  omr  problem.  We,  therefore,  present  several  key  references  for  partially  observed 
Markov  decision  processes.  White  and  White  [45]  give  an  excellent  review  of  standard 
Markov  decision  processes.  They  include  a  short  section  on  extensions  of  MDPs  includ¬ 
ing  that  of  partially  observed  MDPs.  Monahan  [29]  and  Lovejoy  [27]  present  two  good 
surveys  of  POMDP.  Smallwood  and  Sondik  [35]  show  that  for  a  finite  horizon  POMDP 
maximization  problem  the  objective  function  is  piece-wise  linear  and  convex  in  the  cur¬ 
rent  state  probabilities.  They  then  provide  a  dynamic  programming  algorithm  that  uses 
this  property  to  efficiently  solve  the  problem.  Sondik  [36]  later  presents  a  policy  itera¬ 
tion  technique  for  a  discounted,  infinite  horizon  problem.  White  and  Sherer  [43]  present 
three  algorithms  for  the  POMDP  that  are  faster  than  that  of  Smallwood  and  Sondik. 
However,  the  problem  sizes  are  very  limited  in  their  study.  Lovejoy  [24]  [26]  presents 
monotonicity  results  and  provides  feasible  bounds  for  certain  POMDP  problems.  White 
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and  Sherer  [44]  propose  a  heuristic  in  which  only  the  M  most  recent  observations  and  ac¬ 
tions  are  used  to  make  the  cmrent  decision.  They  also  introduce  the  concept  of  ergodicity 
and  discuss  how  this  factor  indicates  the  potential  value  of  historical  information. 

4.3  Optimal  Control  Model  and  Insights 

In  this  section,  we  summarize  the  optimal  control  model  and  discuss  insights  gained 
from  its  structure.  Demand  each  period  arises  from  one  of  a  finite  collection  of  N 
probability  distributions.  The  decision  maker  may  not  know  which  of  the  distributions 
generates  the  demand  in  a  given  period.  Furthermore,  the  distribution  randomly  changes 
from  one  period  to  the  next  according  to  a  known  transition  probability  matrix.  In  this 
chapter,  we  assume  that  the  planning  horizon  is  infinite  with  discounted  costs,  stockouts 
are  always  backordered,  the  order  lead  time  is  either  zero  or  a  known  positive  constant, 
and  there  is  no  fixed  order  cost.  We  define  in  Table  4.1  the  notation  that  will  be  used 
to  describe  the  model. 

Although  we  specify  linear  holding  and  stockout  costs  in  this  basic  model,  many 
of  the  results  hold  for  convex  costs.  The  demand  state  process  {dt}  is  not  known  with 
certainty  and  is  referred  to  as  the  core  process.  It  is  modeled  as  a  finite  state  Markov 
chain,  dt  £  {1,  •  •  • ,  N},  governed  by  the  transition  matrix  P.  If  dt  =  j,  the  probability 
distribution  for  demand  in  t  is  rj,  i.e.,  P[wt  =  k\dt  =  j]  =  rjk  for  fc  =  0, 1, . . . ,  M.  We 
assume  all  cost  factors  and  transition  equations  do  not  change  over  time;  therefore,  the 
optimal  policy,  6,  will  also  be  stationary.  The  core  process  is  partially  observed  through 
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N 

= 

number  of  distributions, 

M 

= 

maximum  demand, 

S 

=z 

stock  level, 

h 

= 

linear  holding  cost, 

V 

linear  stockout  penalty  cost. 

c 

linear  procurement  cost, 

p 

= 

discount  rate. 

1 

lead  time. 

dt 

= 

demand  distribution  in  effect  for  period  t, 

Pij 

= 

P[dt+i  =  j\dt  =  i], 

P 

= 

ipij),  the  transition  probability  matrix  for  dj. 

Wt 

= 

demand  realization  in  period  t. 

= 

lead  time  demand  realization  from  period  t  -^t  +  l  , 

z= 

P[wt  =  k\dt  =  j], 

'^jkl 

= 

P[wt,i  =  k\dt=j], 

It 

vector  of  information  available  up  to  period  t, 

= 

P[dt  =  i\It], 

n 

= 

Xt 

= 

inventory  level  at  the  beginning  of  period 

Ut 

= 

inventory  position  at  the  beginning  of  period 

Ut 

order  quantity  in  period 

Ut,l 

vector  of  outstanding  orders, 

s 

= 

stationary  policy. 

Table  4.1:  Notation 


the  measurement  process  {wt},  and  the  vector  of  information  available  at  time  t  is  /<  = 
Also  known  are  itt  and  ut  where  ttq  is  the  initial  prior  distribution 
that  is  externally  specified.  Rhenius  [32]  has  shown  that  the  prior  distribution,  ttj,  is 
a  sufficient  statistic  for  It  which  implies  that  the  problem  can  be  viewed  as  a  Markov 
decision  process  on  the  state  space  (ut,  tt*).  The  vector  ttj  characterizes  the  current 
belief  of  the  distribution  of  dt  given  all  prior  observations  of  the  information  process 
{tut}.  The  system  dynamics  for  our  CPOMDP  model  are  described  in  Figmre  4.1.  The 


100 


Figure  4.1:  Control  Process 


control  process  begins  with  an  information  vector,  /*,  which  includes  a  prior  distribution, 
TTt,  for  period  t,  and  then  selects  an  order  quantity,  at-  Because  any  optimal  policy,  <5*, 
is  stationary,  the  order  quantity  at  time  t  is  a  function  of  the  current  prior,  TTt,  and 
inventory  position,  ut-  For  a  positive  lead  time,  an  order  placed  I  periods  in  the  past  is 
received.  The  demand,  wt,  then  occurs,  and  the  inventory  costs  for  period  t  cue  incurred. 
The  time  index  advances  to  t+1,  and  the  core  process  advances  from  dt  to  dj+i  according 
to  the  transition  matrix  P.  The  new  prior  distribution  -Kt+i  is  computed  using  ‘Kt,Wt 
and  P.  The  transition  equation  for  the  prior  is 


TTj.t+l  =  Tiinlwt  =  k)  = 


En 
j=l 


(4.1) 


The  problem  is  formulated  in  terms  of  the  inventory  position,  which  is  the  net  inventory 
level  (on-hand  less  backorders)  plus  inventory  on-order  at  the  beginning  of  t,  ut  =  xt  + 
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at-n-  The  inventory  position  follows  the  transition  equation  U(+i  =  ut  +  at  —  wt- 
For  an  inventory  level  v,  define  the  single  period  expected  inventory  cost  function  as 

Gt{v\Trt,l)  =  -  u}  +  hmax{0,v  -  Wt,;}],  (4.2) 

where  wt^i  =  This  cost  function  accounts  for  the  positive  lead  time  in 

computing  the  expected  inventory  costs  for  the  inventory  level  at  time  t+l.  The  optimal 
infinite  horizon  cost-to-go  function  satisfies  the  well  known  Bellman’s  equation  [5]: 

J*(«,7r)  =  -cu  +  min{c5  +  G{S\nJ)  4-  l3E^\^[r{S  -  «;,T(7r|u;))]}.  (4.3) 

In  this  equation,  w  is  the  lead  time  demand  given  the  current  prior,  tt.  A  state  dependent 
base  stock  control  policy  is  optimal  for  this  problem.  Treharne  and  Sox  [40]  demonstrate 
that  a  base  stock  policy  is  optimal  for  the  finite  horizon  problem  because  the  cost  function 
is  a  piecewise-linear  and  convex  (pwl-convex)  function  of  the  inventory  level/position.  A 
simple  extension  of  this  proof  shows  that  a  base  stock  policy  is  optimal  for  the  infinite 
horizon  problem  also.  This  form  suggests  that  J*{u^  tt)  might  be  solved  using  a  standard 
policy  iteration  or  value  iteration  procedure.  Although  the  state  space  on  w  is  a  finite  set, 
the  state  space  on  tt  is  uncountably  infinite.  Therefore,  some  approximation  procedure 
is  necessary  for  most  problems. 

This  problem  has  some  very  useful  properties.  Smallwood  and  Sondik  [35]  show 
that  for  a  finite  horizon  problem  the  optimal  cost  function  is  a  piecewise-linear  and 
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concave  (pwl-cc)  function  of  tt.  Although  their  problem  was  for  a  partially  observed 
Markov  decision  process,  the  property  extends  to  our  problem  which  is  a  composite- 
state,  partially  observed  Markov  decision  process.  For  a  given  inventory  position,  u, 
the  cost  function,  J*(u,7r),  for  the  CPOMDP  is  also  a  pwl-cc  of  tt  when  the  horizon 
is  finite.  However,  in  our  CPOMDP  problem,  the  horizon  is  infinite.  We  conjecture 
that  for  a  given  inventory  position,  u,  the  optimal  cost-to-go  function  will  be  concave 
in  TT,  but  not  necessarily  piecewise-linear.  The  cost  function  is  easier  to  compute  if  it 
is  piecewise-lineax.  Sondik  [36]  shows  that  the  cost  function  for  a  partially  observable 
Markov  decision  process  is  piecewise-linear  if  the  optimal  policy  belongs  to  the  class  of 
finitely  transient  policies.  Not  all  policies  have  this  property,  and  additionally,  there 
are  no  simple  rules  for  determining  if  a  policy  is  finitely  transient.  Because  the  cost 
function  for  our  problem  is  concave  in  tt,  we  can  approximate  the  optimal  policy  by  a 
finite  number  of  subgradients  to  the  cost  function.  We  will  show  in  Section  4.4  how  these 
subgradients  can  be  computed. 

4.4  Suboptimal  Control 
4.4.1  Grid  Approximation 

To  evaluate  these  policies,  we  would  ideally  compare  the  suboptimal  policies  to 
known  optimal  results.  Unfortunately,  because  the  state  space,  H,  is  uncountably  infinite, 
we  cannot  do  so.  Therefore,  we  have  chosen  to  approximate  the  optimal  cost  function 
using  a  grid  approximation  method.  We  do  so  by  discretizing  the  state  space  on  II. 
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Using  G  +  \  grid  points  for  tt^  such  that 


f  ^ 

n=<^7r^ei?":^7rf  =  1,  Trf 
I  i=l 


(4.4) 


We  index  the  element  of  each  set  as  .  It  is  necessary  to  calculate  the  probabiUty  of 
transitioning  from  one  allowable  discrete  state  in  ft  to  another  discrete  state  in  ft.  The 
updated  prior,  TTt+i  =  Ti{'Kt\wt  =  A:),  will  most  likely  not  be  a  feasible  value  of  tt,  that 
is,  it  will  not  be  a  member  of  ft.  Thus,  we  find  the  nearest  feasible  grid  points  in  ft  that 
contain  -Kt+i-  We  then  calculate  the  transition  probability  that  the  updated  prior  will 
move  to  a  particular  feasible  value  of  tt  in  the  grid.  The  baxycentric  coordinates  of  the 
updated  prior  with  respect  to  these  nearest  feasible  grid  points  represent  the  transition 
probabilities  to  these  points.  Lovejoy  [26]  provides  details  and  demonstrates  an  efficient 
way  to  calculate  these  barycentric  coordinates  using  a  Ereudenthal  Triangulation  [9] 
technique  and  a  transformation  of  the  state  space,  ft.  This  triangulation  technique  allows 
any  point  in  the  state  space  to  be  quickly  triangulated.  However,  if  the  triangulation 
is  done  only  in  ft,  then  the  resulting  barycentric  coordinates  may  not  be  at  feasible 
vertices  in  ft.  We  briefly  review  these  procediures  in  the  context  of  our  problem.  Given 
an  updated  prior,  we  must  estimate  the  probabilities  that  the  updated  state  in  ft 
will  be  at  a  finite  number  of  feasible  vertices. 

Calculation  of  updated  state  space  in  ft: 
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Define  a.n  N  x  N  matrix: 


1.  Choose  a  grid  spacing  with  G  intervals. 

2.  For  any  posterior  probability,  calculate  the  coordinates  of  x  in  transformed 

state  space  where  x  =  Note  that  has  a  simple  form;  it  is  an  upper 

triangle  matrix  with  all  elements  equal  to  G. 


3.  Determine  vertices  that  contain  x  using  Preudenthal  Triangulation.  The  base  ver¬ 
tex,  vij  is  equal  to  the  largest  integer  vector  less  than  or  equal  to  x.  Calculate  the 
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direction,  d,  from  the  base  vertex,  vi,  to  x,  d  =  x  -vi.  Order  the  components  of  d 
in  descending  order  with  p  being  the  permutation  of  the  integers  {1, . . .  ,n}  that 
indicates  the  ordering  {dpi  >  dp2  >  ■ .  ■  >  dpn).  Define  ej  as  the  unit  vector. 
The  remainder  of  the  vertices  that  form  the  triangulation  are 


'02  =  V\+  Cpi 


Vz  —  V2'\‘  ep2 


V4  —  Vz  CpZ 


-Vk  +  Cpk  k<  n. 

4.  Determine  the  barycentric  coordinates  Aj  using  Preudenthal  Triangulation. 

~  dpji 

\ji  —  ^pn 

'^n— 1  —  ^p(n— 2)  ^p(n— 1) 

A2  ”  dpi  dp2 
nH-1 

Xi  =  i-J2k 

2=2 
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5.  Transform  the  vertices  with  positive  Aj  back  to  state  space  H. 

TT^  =  B  *V^ 

TT^  =  B  *V^ 

=  B* 

The  transition  probability  from  tt^+i  to  =  Xj. 

Consider  the  following  example.  Assume  that  iV  =  4  distributions,  and  that  G  =  10 
which  is  spacing  =  .01.  Determine  the  vertices  and  transition  probabilities  if  the  updated 
prior,  TTt+i  =  [.48,  .28,  .13,  .11].  First,  x  =  B~^Trt+i  =  [10, 5.2, 2.4, 1.1]. 

1.  Determine  the  vertices  that  triangulate  x. 

(a)  ui  =  [10,5,2,l]  d=[0,.2,.4,.l]  p=[3,2,4,l] 

(b)  V2  =  [10, 5, 2, 1]  +  [0, 0, 1, 0]  =  [10, 5, 3, 1] 

(c)  V3  =  [10, 5, 3, 1]  +  [0, 1, 0, 0]  =  [10, 6, 3, 1] 

(d)  t;4  =  [10, 6, 3, 1]  +  [0, 0, 0, 1]  =  [10, 6, 3, 2] 

(e)  V5  =  [10, 6, 3, 2]  +  [1, 0, 0, 0]  =  [11, 6, 3, 2] 


2.  Determine  the  barycentric  coordinates  for  x  relative  to  vertices. 
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(a)  As  =  0.0 

(b)  A4  =  0.1  -  0.0  =  0.1 

(c)  As  =  0.2  -  0.1  =  0.1 

(d)  A2  =  0.4  -  0.2  =  0.2 

(e)  Ai  =  1  -  0.2  -  0.1  -  0.1  -  0.0  =  0.6 

3.  Vertices  transformed  to  fl. 

(a)  ■k^  =  B*v^  =  [0.5, 0.3, 0.1, 0.1] 

(b)  n^  =  B*v‘^  =  [0.5, 0.2, 0.2, 0.1] 

(c)  tv^  =  B*v^  =  [0.4, 0.3, 0.2, 0.1] 

(d)  =  =  [0.4, 0.3, 0.1, 0.2] 

(e)  tt^  =  B*v^  =  [0.5, 0.3, 0.1, 0.2] 

First,  note  that  tt®  is  infeasible.  However,  A5  =  0.0,  and  there  is  no  transition  probability 
to  TT^.  Therefore,  if  the  updated  prior  is  TTt+i  =  [.48,  .28.13,  .11],  we  estimate  a  transition 
probability  of  0.6, 0.2, 0.1,  and  0.1  to  feasible  vertices  7r^,7r^,7r^,  and  respectively. 
The  procedures  to  find  the  feasible  vertices  and  respective  transition  probabilities  are 
straightforward  to  implement.  B~^  is  simple  to  calculate,  and  both  the  vertices  and 
barycentric  coordinates  are  computationally  easy  to  determine. 
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The  linear  approximation  allows  us  to  represent  every  feasible  transition  in  the  state 
space.  Using  this  grid  approximation  and  linear  interpolation,  the  cost-to-go  function  is 

Jg{u^  =  mini  cmax(0,  S  —  u)  G(max(ii,  5)|7r^,  1) 

S^O  I 

'N(G+1) 

+  Xj{T{Tr'‘\w))JG{rciax{u,S)-w,Tr^) 

j=l 

where  \j{T{Tr'^\w))  axe  the  coordinates  of  the  Preudenthal  triangulation  of  T[tx'^\w)  over 
the  grid  ft.  Therefore,  for  each  grid  point  and  each  possible  value  of  w,  we  must 
compute  the  Preudenthal  Triangulation  of  T{-k^\w).  Finally,  the  cost  for  any  value  of  tt 
can  be  computed  by 

G+1 

Jg{u,  tt)  =  ^  XiJaiu,  B  *  Vi)  (4.6) 

i=l 

Lovejoy  [26]  shows  that  Jg{u,  tt)  is  also  a  lower  bound  to  the  optimal  cost  function. 
4.4*2  Upper  Bound 

We  next  extend  the  upper  bounding  procedure  described  in  Lovejoy  [28]  for  this 
POMDP  problem.  In  the  stationary  problem  studied  in  Lovejoy  [28],  if  the  partially 
observed  state  becomes  known  with  certainty,  the  problem  is  then  fully  observed  and  is 
solvable  by  conventional  dynamic  programming  algorithms.  Van  Hee  [41]  exploits  this 
fact  to  show  how  upper  and  lower  bounds  can  be  developed  by  solving  fully  observed 
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problems.  This  characteristic  of  the  stationary  problem  also  means  that  Lovejoy’s  upper 
bound  is  tight  at  the  extreme  points.  In  the  case  of  a  non-stationary  problem,  even  if  we 
know  the  current  state  with  certainty,  we  will  not  know  the  future  states  with  certainty. 
Therefore,  it  is  not  clear  how  Lovejoy’s  procedure  can  be  effectively  extended  to  the  case 
of  a  non-stationary  process. 

In  order  to  develop  our  upper  bounding  procedure,  we  first  consider  the  class  of 
static  base  stock  policies.  A  static  base  stock  policy  chooses  a  base  stock  level  S  given 
an  initial  prior,  tt,  and  uses  that  same  base  stock  level  in  all  future  periods.  Such  a 
policy  is  clearly  suboptimal,  and  as  such  it  provides  an  upper  bound  for  the  optimal  cost 
function.  For  an  initial  inventory  position  u,  an  initial  prior  distribution  tt,  and  a  base 
stock  level  S,  the  infinite  horizon  discounted  cost  is  given  by 

J{u,  7r,S)  =  c  max(0, 5— u)-f-G(max(u,  S'jjTr,  l)+/3E^^j^[J{Tnax{u,  S)—w,  T(7r|u;),  5)]. 

(4.7) 

Because  this  policy  does  not  use  information  about  the  future  values  of  tt.  Equation  (4.7) 
can  be  re-written  as 

N 

J{u,n,S)  =  cm.ax(0,S  —  u)  -I-  ^7riG{max{u, S)\ei,l) 

1=1 

N  N 

PijE^\e.[J{max.{u,  S)  -  w,  Cj,  5)]  (4.8) 

1=1  j  =  l 


no 


which  is  readily  computed  as  the  solution  of  a  system  of  linear  equations  for  a  fixed  value 
of  TT.  Also,  note  that  J(u,  tt,  S)  is  linear  in  tt.  The  infinite  horizon  discounted  cost  of  the 
optimal  static  base  stock  policy  is 

Js{u,Tr)  =  xmnJ{u,7r,S)  (4.9) 

which  is  piecewise  linear  and  concave  in  tt  because  J{u,  tt,  S)  is  a  linear  function  of  tt  for 
each  S.  Define  Si  =  argmin5>o  J(u,ei,5)  which  is  the  optimal  static  base  stock  level 
given  that  demand  distribution  i  is  known  to  be  the  initial  demaud  distribution,  i.e., 
Js{u, Si)  =  J{u,  Ci,  Si).  For  j  =  1, . . .  ,N,  define  the  vector  ^(u)  as 


Tpl{u)  =  J{u,ei,Sj)  =  cmax(0, —  u)  +  G{max{u,  Sj)\ei,l) 


N 


+  /3  ^  PikEy,]ei  [J (max(u,  Sj)  -  w,  e*,,  S'j)].  (4.10) 


A;=l 


Prom  Equations  (4.8)  and  (4.9),  we  see  that  ?/?^(u)  is  a  subgradient  of  J5  at  the  point 
(u, ej)  because  Js{u^ej)  =  J(u,ej,iSj)  and 


N 


J{u,TT,Sj)  =Y^TT, 


2=1 


N 


cmax(0,  Sj  —  u)  +  G(max(u,  S'j)|et,  1) 
+  ^22  [«^(max(u,  Sj)  -  w,  ek,  5^)] 

k=l 

N 


2=1 


(4.11) 


Ill 


Furthermore,  we  have  that 

J(u,  TT,  Sj)  >  Jsiu,  tt)  >  riu,  n)  for  all  j  =  1, . . .  ,  N. 


Therefore, 

N 

J{u,n)=  min  y^7rt^(u)  >  Js{u,ir,S)  >  J*(u,  tt).  (4.12) 

.^1 

The  function  7{u,n)  is  also  a  piecewise  linear  function  of  tt.  Because  it  is  based  on 
a  set  of  static  base  stock  policies,  it  may  admittedly  be  a  weak  bound  in  some  cases. 
Therefore,  we  would  like  to  improve  this  boimd  at  values  of  n  where  it  is  not  tight.  Such 
a  refinement  can  be  accomplished,  as  in  Lovejoy  [28],  by  using  one  or  more  dynamic 
programming  iterations  at  the  chosen  point(s).  Define  l{t)  =  argmin^ 
the  one-step  DP  recursion  is 

tt)  =  HJ{u,n)  =  ^n|cmax(0,  S  —  u)  +  G(max(u,  5)|7r,Z) 

+  S)  -  ty,r(7r|u;))]  |.  (4.13) 


As  demonstrated  by  Lovejoy  [28], 


J(u,7r)  >  J^{u,-k)  >  J*(u,7r). 
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Note  that  the  expected  future  cost  in  Equation  (4.13)  can  be  expressed  as 


N  M  N  _ 

E  E  E ' "t;  ’  •  S)  -  k) 

i=l  k-0  j=l  ^n=l  '^n’^nk 


M  N  N 

=  EEE--^  nkPnj  •  (max(u,  S)  -  k) 

k=Q  j=l  n=l 


N  M  N 

=  '^n'^rik'^ Pij  •  (max(u, S)  -  k) 

t=l  fc=0  j=l 

which  is  piecewise  linear  and  concave  in  tt.  Therefore,  J^{u,'k)  is  piecewise  linear  and 
concave  in  tt.  Let  j  =  N  +  1  and  define 


=  cmax(0,5jv+i  —  «)  +  G(max(u,5;v’+i)|ei,0 

M  N 

+  /3  ^  ^  (max(u,  5iv+i  -  k)  (4.14) 

fc=0  j=i 

for  some  tt.  Then,  given  by  Equation  (4.14)  is  a  subgradient  of  at  (u, tt). 

Furthermore,  J(it,  tt)  =  niinj=:i,...  ,7\r+i  iTiV’l  (^i)  >  J^(n,  tt)  >  J*(tt,  tt).  Note  that 
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when  P  =  /,  as  in  Lovejoy  [28],  Equation  4.14  simplifies  to 

M 

=  cmax(0,5’  -  u)  +  G{max{u,  S  -  k). 

k=o 

(4.15) 

Thus,  our  procedure  is  to  first  initialize  the  upper  bound  function  J{u,'k)  using 
Equation  (4.10)  for  j  =  1, . . .  ,  JV.  Then  for  some  selected  values  of  tt,  say  tt^,  . . .  ,7r^, 
Equation  (4.14)  is  used  to  generate  additional  subgradients  ...,  The  upper 

bounding  function  is  given  by  J(u,7r)  =  minj=ij...,iV+A’idI:i  7rii/>|  (u).  This  process  can 
be  repeated  to  add  additional  subgradients  to  the  definition  of  J(u,  tt). 

It  should  be  noted  that  the  upper  and  lower  bounds  are  tight  at  the  extreme  point 
for  the  stationary  problem.  In  the  non-stationary,  there  is  a  gap  between  these  bounds. 
Figure  4.2  shows  the  gap  between  these  bounds.  The  improvement  steps  discussed 
tighten  these  bounds. 

4.4.3  Lower  Bound 


We  now  present  a  lower  bounding  procedure. 
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Figiire  4.2:  Comparison  of  Stationary  and  Non-Stationary  Bounds 

1.  Initialize  the  lower  bound  by  solving  the  fully  observed  problem  at  the  extreme 
points  of  n,  Ci,  and  let  Cj)  =  value  at  Cj,  where 


N 


J°(u,  Bi)  =  min  ^  c{Si  -  u)  +  G{Si\d  =  i,l)+py]  E^\a=i[PijJ°iSi  -  w,  e^)] 

I  - 

J  =  1 


(4.16) 


2.  Estimate  J  at  a  fixed  set  of  grid  points  using  a  convex  combination  of  {J^(u,ei)}: 

N 

J°{u,Tr’‘)  =  ^Trf  >(u,ei). 

2=1 
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3.  Use  a  sequence  of  DP  recursions  to  update  values  at  the  fixed  set  of  grid  points  so 
that 


J"(u,7r*')  =  mn<  cmax(0,  S  —  u)  +  G(max(tt,  5)|7r*^,  Z) 


'N(G+1) 

Aj(r(7r*|t«))J”“^(max(u,  S')  —  w,7r^) 

j=i 


►  ,  for  n  =  1, 2, . . .  , 


where  {Aj(T(7r*^|w))}  are  the  coordinates  of  the  Preudenthal  triangulation  of 
T(7r^|n;)  over  grid  G. 

We  mahe  the  following  two  conjectures  which  assume  that  the  same  II  is  used  for  both 
Jq  and  J”  V  n. 

Conjecture  1.  J”(u,7r*^)  <  V  n,u,  and  tt*'. 

Conjecture  2.  limn^oo  V  u,  and  tt*. 


4.4.4  Myopic  and  Limited  Look-Ahead  Policies  (Infinite  Horizon) 
Define 


J^°(u,7r)  =  mn{c(S  —  u)  +  G(S|7r,Z)} 


(4.17) 


as  the  myopic  cost  function  which  can  be  re-written  as 


J^°(u,  tt)  =  g^in  TTj  j^c(S  -  u)  -f  G(S|et,  1)  |.  (4-18) 
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Therefore,  J^^{u,k)  is  piecewise  linear  and  concave  in  tt.  Let 

o:{(u)  =  cmax(0,  Sj  —  u)  +  G(max(u,  Sj)\ei,l), 

then 

N 

tt)  =  min  y^7ria|(ti).  (4-19) 

Because  is  a  linear  function  of  tt,  tt)  is  piecewise  linear  and  concave 

in  TT. 

The  one-period  look-ahead  cost  function  is 

c{S  -u)  +  GiS\n,  1)  +  {S  -  w,  r(7r|u;))  |  (4.20) 

which  can  be  re-written  as 

tt)  =  min 
^  ^  S>u 

(4.21) 

where  l{T{Tr\w))  =  argminj  Ti{TT\w)o^^  (S—w).  Therefore,  tt)  is  also  piecewise 

linear  and  concave  in  tt.  Cheng’s  [7]  linear  support  algorithm  can  be  used  to  identify  all 
the  subgradients  needed  to  define  tt)  =  minj  (^)- 


117 


1.  Calculate  the  subgradient  7^(n)  at  each  extreme  point,  ej,  of  11  and  store  in  the 
set  Gt{u). 

2.  For  each  'y^{u)  6  Gt{u),  identify  an  extreme  point,  of 

R{'Y^{u))  =  {7r|7r^7-'(ti)  <  7r^7*(u)V  I  ^  j} 
by  solving  for  some  j 

min  7r^7*^(it) 

s.t.  7r^{'y^u)  —  7*(n))<  0  'i  I  ^  j 
J^7ri  =  l 
TTi  >  0  Vi 

3.  Define  =  minj  7r^7'^(«),  and 

J^^{u,Tr)  =  min/c(S'  —  n)  +  C(5|7r,  1)  +  —  tu|r(7r|n;))]  1. 

S^u  ^  J 

If  J(u,7r-^*)  >  then  compute  the  subgradient,  7(n),  of  at  and 

add  to  Gt{u),  and  go  to  2.  Otherwise,  next  k  or  next  j  or  end. 
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Initial 

Truncated 

Dist. 

Mean 

Std.  Dev. 

Mean 

1 

2.00 

1.5 

2.00 

2 

14.00 

4.0 

13.45 

Table  4.2:  Demand  Distributions 


Discount 

=  .9 

Leadtime 

=  0 

Critical  Ratio  =  0.727 

Assume  observed  lost  sales 

Table  4.3:  Experiment  Parameters 


4.5  Example  Case 

In  this  section  we  present  an  example  of  how  to  determine  an  approximation  using 
the  grid  technique.  We  also  show  the  subgradients  that  form  an  upper  bound  to  the  op¬ 
timal  cost  function.  There  are  two  distributions  that  form  the  set  of  possible  solutions. 
Each  of  the  distributions  were  initially  generated  using  negative  binomial  distributions 
and  then  truncated  at  a  maximum  demand  of  18  and  rescaled.  Table  4.2  shows  the  de¬ 
mand  parameters.  The  second  and  third  columns  show  the  mean  and  standard  deviation 
before  truncation,  and  the  foiurth  column  shows  the  mean  of  the  truncated  distributions. 


Cost 


9 


Figure  4.4: 


mate  Costs  and  Order  Quantities  at  tti 
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Figure  4.5:  Cost-togo  function  at  various  inventory  levels 


The  transition  matrix  is 


P  = 


(  0.8  0.2 


\ 


0.2  0.8  J 


The  number  of  grid  points  determines  the  accuracy  of  the  approximation.  In  Fig¬ 
ure  4.3  we  show  a  cost  function  with  7  different  levels  of  grid  spacing,  A  =  ^.  This 
example  has  two  distributions,  so  the  x-axis  represents  the  value  of  tti.  For  A  =  .25,  .33, 
and  .50,  we  show  the  linear  approximation  of  the  cost  function.  Notice  that  the  cost 
function  appears  to  converge  very  quickly  once  the  grid  size  is  reduced  to  .10  and  smaller. 
This  technique  can  serve  as  the  actual  approximation  of  the  cost  function.  For  example, 
the  same  case  is  shown  in  Figure  4.4  with  A  =  .02.  The  right  vertical  axis  shows  the 
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Figure  4.6:  Initial  Upper  Bound 

order-up- to  quantities  (ranging  from  3  -  15)  at  each  allowable  value  of  tti.  In  Figure  4.5 
we  show  the  cost-to-go  function  at  all  possible  inventory  levels.  A  much  higher  penalty 
is  incurred  when  the  inventory  level  is  high  and  there  is  a  strong  belief  (tti  close  to  1) 
that  the  current  demand  distribution  is  the  low  demand. 

In  Figure  4.6  we  show  the  optimal  cost  function  from  the  grid  approximation.  We 
also  show  the  first  two  subgradients  that  form  the  upper  bound  on  this  function.  The 
first  two  subgradients  were  formed  by  evaluating  the  costs  at  the  extreme  points.  The 
initial  two  subgradients  were  formed  with  static  polices  of  11  and  14  respectively.  In 
Figure  4.7  we  show  five  additional  subgradients  which  were  found  by  performing  single 
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Figure  4.7:  Upper  Bo 
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Subgradient 

Recursion 
Point  (tti) 

Cost  at  TTi  =  1 
(Low  demand) 

Cost  at  TTi  =  0 
(High  Demand) 

Action 

1 

248.71 

236.88 

11 

2 

265.11 

212.39 

14 

3 

msHi 

254.17 

214.56 

13 

4 

257.22 

211.98 

14 

5 

262.65 

5 

6 

219.37 

3 

7 

0.0 

262.12 

209.91 

15 

Table  4.4:  Initial  Subgradients  and  Actions 


Lower  Bound 

Grid  Spacing 

.50 

.33 

.25 

.10 

.05 

.02 

.01 

Run  Time 

0.2 

0.5 

0.8 

4.1 

Min 

9.09 

2.61 

0.72 

0.04 

0.01 

0.01 

0.01 

Max 

14.70 

4.63 

2.15 

0.43 

0.25 

0.19 

0.19 

Avg 

11.72 

3.60 

1.18 

0.10 

Table  4.5:  Lower  Bound  Run  Times  and  Percent  Deviation  from  UB 


dynamic  recursions  at  tti  =  .5,  .25,  .75, 1.0,  and  0.0.  Table  4.4  shows  the  values  of  the 
first  seven  subgradients  and  their  respective  order-up-to  levels.  If  we  perform  a  recursion 
at  TTi  =  .5,  the  resulting  subgradient  is  based  on  an  action  of  13.  We  would  expect  it  to 
be  in  the  range  between  11  and  14.  If  we  are  fairly  confident  that  the  demand  is  high 
(tti  =  .25),  then  we  would  expect  the  next  subgradient  to  be  based  on  a  high  stock  level 
which  coincides  to  its  actual  value  of  14.  However,  if  the  demand  is  thought  to  be  low, 
we  order-up-to  5.  If  we  are  certain  the  demand  is  low,  the  initial  stock  level  is  3.  We 
can  continue  to  add  subgradients  in  this  manner  to  improve  the  upper  bound. 
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Upper  Bound 

Recursions 

0 

5 

20 

50 

100 

150 

200 

250 

300 

500 

Rrm  Time 

■a 

0.9 

1.6 

3.5 

7.8 

14.2 

21.9 

34.0 

43.5 

1:46.3 

12.00 

8.20 

0.18 

0.08 

0.04 

.01 

Max 

35.01 

19.73 

11.59 

4.46 

0.79 

0.41 

0.29 

0.23 

.19 

Avg 

19.01 

15.41 

9.20 

3.79 

1.26 

0.53 

0.24 

0.11 

0.06 

.02 

Table  4.6:  Upper  Bound  Run  Times  and  Percent  Deviation  from  LB 


Tables  4.5  and  4.6  shows  the  run  times  on  a  SUN  workstation  to  calculate  a  lower 
bound  using  grid  approximation  and  an  upper  boimd  using  the  subgradient  techniques. 
Table  4.5  shows  the  deviation  from  the  upper  bound  (500  recursions)  of  the  lower  bound 
at  various  grid  spacings.  At  a  grid  spacing  of  .10  and  smaller  the  cost  function  converges 
very  rapidly.  The  solution  at  this  spacing  is  computed  in  only  4.1  seconds.  Table  4.5 
shows  the  deviation  of  each  upper  bound  from  the  grid  approximation  from  the  best 
lower  bound  (spacing  =  .01).  Notice  that  after  150  recursions  that  the  percent  deviation 
is  less  than  one  percent  at  every  grid  point.  This  is  accomplished  in  14.2  user  seconds. 
This  suggests  that  the  upper  bound  subgradient  technique  can  be  a  very  valuable  and 
efficient  procedure  to  get  a  tight  upper  bound  on  the  cost  function. 

4.6  Conclusion 

In  practice,  demand  is  frequently  non-stationary  and  also  characterized  by  partial 
information.  These  two  characteristics  significantly  complicate  the  problem.  Therefore, 
many  practitioners  simplify  the  problem  by  assuming  either  that  the  demand  is  sta¬ 
tionary  or  that  the  underlying  demand  distribution  is  completely  known.  Neither  of 
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these  assumptions  is  necessary.  We  show  that  the  optimal  cost  function  is  pwl-cc.  This 
characteristic  allows  us  to  use  a  grid  approximation  method  to  estimate  the  optimal  cost 
function.  This  approximation  may  be  computationally  difficult  to  solve  if  the  state  space 
is  large.  We  show  two  other  techniques  to  compute  both  an  upper  and  a  lower  bound 
to  the  optimal  cost  function.  These  bounds  are  not  as  tight  as  the  bounds  generated  in 
a  stationary  problem.  However,  we  may  iteratively  improve  both  the  upper  and  lower 
bounds.  These  bounds  can  be  used  to  produce  policies  that  provide  significant  cost  sav¬ 
ings  to  the  decision  maker.  Finally,  we  present  myopic  and  look-ahead  policies  which 
also  can  be  used  to  approximate  the  cost  function. 


Chapter  5 


Conclusions  and  Future  Research 

Our  goal  was  to  show  that  problems  with  non-stationary  demajid  and  partial  in¬ 
formation  can  be  solved  by  efficient  and  effective  procedures  without  making  commonly 
used  and  costly  assumptions.  We  have  done  so  by  first  reviewing  key  literature  and 
then  presenting  a  basic  model  and  reviewing  optimal  policies  and  their  characteristics 
for  both  censored  and  uncensored  stationary  demand  with  partial  information.  We  also 
discuss  cases  in  which  state  space  reductions  make  problems  much  easier  to  solve.  We 
also  note  that  most  realistic  cases  will  require  some  type  of  suboptimal  policy.  We  then 
presented  a  model  for  the  case  of  non-stationary  demand  with  partial  information.  We 
model  the  problem  as  a  composite-state,  partially  observed  Markov  decision  process. 
When  polices  use  feedback  and  consider  all  the  demand  uncertainty  over  some  period 
of  time,  performance  which  clearly  out-matches  standard  CEC  approaches  can  easily  be 
obtained.  We  provide  limited  look-ahead  and  open-loop  feedback  control  models  and  test 
these  policies  against  standard  CEC  approaches  as  well  as  optimal  approaches.  Next,  we 
presented  the  same  non-stationary  demand  problem,  although  with  an  infinite  planning 
horizon.  In  this  case,  the  optimal  cost  function  will  usually  not  be  computable  due  to  the 
intractability  of  the  state  space.  Therefore,  we  develop  a  grid  approximation  technique 
that  is  valid  for  this  problem.  We  then  show  how  to  calculate  and  improve  both  an  upper 
and  a  lower  bound  for  the  optimal  cost  function. 
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Non-stationary  demand  is  common  in  many  production  control  environments.  Ad¬ 
ditionally,  despite  a  wealth  of  information,  these  same  environments  will  often  be  char¬ 
acterized  by  partial  information  for  the  demand  state.  Managers  can  solve  many  of  these 
problems  by  techniques  presented  in  this  dissertation  and  obtain  significant  monetary 
savings  over  policies  commonly  used  today. 

This  research  can  be  expanded  in  several  important  ways.  One  of  the  first  areas 
that  we  would  like  to  expand  this  research  to  is  what  we  call  constrained  inventory 
management  for  a  volatile  demand  process.  This  problem  has  non-stationary  demand 
and  partial  information.  However,  we  will  also  incorporate  a  production  constraint  in  the 
system,  which  exists  in  many  real  life  applications.  We  plan  to  provide  an  optimal  model 
and  a  detailed  discussion  of  its  characteristics.  Because  optimal  policies  are  computable 
for  only  small  problems,  we  will  also  develop  suboptimal  policies.  We  will  test  these 
policies  over  a  wide  range  of  problem  instances  and  compare  them  with  an  optimal 
policy.  Finally,  we  will  show  how  these  procedures  can  provide  large  cost  savings  in  a 
real  world  example. 

In  this  dissertation,  we  developed  models  for  some  fairly  general  problems.  Some 
of  the  analytical  results  apply  to  problems  with  various  assumptions.  We  would  like  to 
conduct  more  computational  testing  to  more  thoroughly  understand  the  nature  of  these 
problems.  For  example,  we  would  like  to  do  more  computational  testing  with  positive 
leadtimes  and  fixed  setup  costs.  We  would  also  like  to  more  closely  examine  Ccises  with 
lost  sales  in  which  the  demand  may  not  be  completely  observed.  Lost  sales  problems  will 
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lead  to  the  concept  of  dual  control  which  balances  cost  minimization  with  the  value  of 
uncensored  observations.  Much  of  this  research  is  dependent  on  an  externally  supplied 
prior  on  the  unknown  parameter.  We  would  like  to  conduct  sensitivity  analysis  on  the 
prior.  This  will  provide  important  insights  into  the  value  of  present  information. 

We  would  like  to  do  more  thorough  testing  of  various  transition  structures.  We  have 
the  capability  to  model  increasing,  decreasing,  cyclical,  and  other  demand  structures. 
More  computational  testing  of  these  situations  may  offer  valuable  insights. 

Finally,  a  major  but  important  extension  of  this  research  would  be  to  extend  this 
work  to  a  multi-echelon  problem.  This  extension  will  give  important  information  relevant 
to  supply  chain  management  and  control. 
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